Subject: [dispatch] Chartering SIP Overload

From: Gonzalo Camarillo <[email-masked]>
Date: Fri, 15 Jan 2010 13:23:04 +0200

Folks,

as you know, we agreed to charter the SIP overload work a while ago. 
Below you can find the proposed charter.

We have gotten some comments back from the IESG. There are some concerns 
about the effects of the proposed solutions in real networks. It seems 
that those concerns would disappear if we chartered the WG to develop 
Experimental proposals, at least at the beginning. Those proposals would 
be able to eventually move to the standards track once we got 
operational experience with them. Comments?

Thanks,

Gonzalo

SIP Overload Control WG Charter
-------------------------------

As with any network element, a Session Initiation Protocol (SIP) server
can suffer from overload when the number of SIP messages it receives
exceeds the number of messages it can process. Overload is said to occur
when a SIP server does not have sufficient resources to process all
incoming SIP messages.  These resources can include CPU, memory, network
bandwidth, input/output, or disk resources.

Overload can pose a serious problem for a network of SIP servers. During
periods of overload, the throughput of a network of SIP servers can be
significantly degraded.  In fact, overload may lead to a situation in
which the throughput drops down to zero or a small fraction of the
original processing capacity.  This is often called congestion collapse.

The objective of a SIP overload control mechanism is to enable SIP
servers to operate close to their capacity limit during times of overload.

The SIP protocol provides a limited mechanism for overload control
through its 503 (Service Unavailable) response code and the Retry-After
header.  However, this mechanism cannot fully prevent overload of a SIP
server and it cannot prevent congestion collapse.  In fact, its on/off
behavior may cause traffic to oscillate and shift between SIP servers
and thereby worsen an overload condition.  A detailed discussion of the
SIP overload problem, the problems with the 503 (Service Unavailable)
response code and the Retry-After header and the requirements for a SIP
overload control mechanism can be found in RFC5390.

The objective of the proposed working group is to develop mechanisms for
SIP overload control. Two complementary approaches exist for handling
overload situations:
- The first mechanism aims at preventing overload in SIP servers by
adjusting the incoming load to a level that is acceptable for this
server. This mechanism uses implicit and/or explicit feedback to
determine when an overload condition occurs in a SIP server and a
reduction in load is required.
- The second mechanism aims at preventing overload in SIP servers by
distributing load control filters to SIP servers that throttle calls
based on their destination domain, telephone number prefix or for a
specific user. Such filters can be used, for example, to throttle calls
to a hotline during a ticket-giveaway event.

Specifically, the proposed working group will develop the following
deliverables:
1. A document describing key design considerations for a SIP overload
control mechanism.
2. A specification for an SIP overload control mechanism based on
implicit/explicit feedback.
3. A specification for a SIP load filtering mechanism.

---

From: Volker Hilt <[email-masked]>
Date: Fri, 15 Jan 2010 09:46:49 -0500

Gonzalo,

it would be great to understand what the specific concerns are.

We had a design team working on simulations on SIP overload control for 
a substantial period of time. The group has produced many results and 
has published these results at scientific conferences and status reports 
to the IETF. Some of the drafts have been implemented and tested in 
prototypes or products by multiple vendors. What else is needed?

I'm worried that we are taking yet another detour which will further 
delay the process.

Volker

Gonzalo Camarillo wrote:
> _______________________________________________
> dispatch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch

---

From: Gonzalo Camarillo <[email-masked]>
Date: Fri, 15 Jan 2010 17:02:02 +0200

Hi,

> I'm worried that we are taking yet another detour which will further 

this would not delay the process in any way. If something, it could 
speed it up, because the requirements to review Experimental RFCs are 
lower than to review standards track RFCs... at the end of the day, I do 
not expect this to influence the time it takes to publish the specs 
because I am sure the group will make sure the quality of the specs as 
high as for a standards-track RFC.

Cheers,

Gonzalo

---

From: Volker Hilt <[email-masked]>
Date: Fri, 15 Jan 2010 10:08:03 -0500

Gonzalo,
> high as for a standards-track RFC.
I'd like to understand the concerns that lead to the proposal of putting 
this work through the experimental track.

What are we missing that prevents us from creating a standards track 
document?

Volker

---

From: Janet P Gunn <[email-masked]>
Date: Fri, 15 Jan 2010 10:18:52 -0500

Is the concern "real networks" vs "simulated networks"?

Or is the concern "the (open) Internet" vs "(closed, walled-garden) 
carrier networks"?

Janet

This is a PRIVATE message. If you are not the intended recipient, please 
delete without copying and kindly advise us by e-mail of the mistake in 
delivery. 
NOTE: Regardless of content, this e-mail shall not operate to bind CSC to 
any order or other contract unless pursuant to explicit written agreement 
or government initiative expressly permitting the use of e-mail for such 
purpose.

From:
Volker Hilt <[email-masked]>
To:
Gonzalo Camarillo <[email-masked]>
Cc:
DISPATCH <[email-masked]>, "[email-masked]" 
<[email-masked]>
Date:
01/15/2010 09:47 AM
Subject:
Re: [sip-overload] [dispatch] Chartering SIP Overload

Gonzalo,

Volker

_______________________________________________
sip-overload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/sip-overload

---

From: Robert Sparks <[email-masked]>
Date: Fri, 15 Jan 2010 09:34:12 -0600

(Copying Lars - please preserve him on the CC for this thread).

The big concern, as I understand it, is that the state of what the  
team's explored so far
reduces the degradation of throughput as load increases, but still  
eventually collapses.
Is that accurate? (I might have missed something the team's done that  
manages to converge
to a stable point.)

RjS

On Jan 15, 2010, at 9:08 AM, Volker Hilt wrote:

> Gonzalo,
>>> I'm worried that we are taking yet another detour which will  
>>> further delay the process.
>> speed it up, because the requirements to review Experimental RFCs  
>> are lower than to review standards track RFCs... at the end of the  
>> day, I do not expect this to influence the time it takes to publish  
>> the specs because I am sure the group will make sure the quality of  
>> the specs as high as for a standards-track RFC.
> I'd like to understand the concerns that lead to the proposal of  
> putting this work through the experimental track.
>

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 15 Jan 2010 10:38:24 -0500

At least according to our measurements and simulations, that is not the case. The throughput stays at ~100% until infinity.

Henning

On Jan 15, 2010, at 10:34 AM, Robert Sparks wrote:

> The big concern, as I understand it, is that the state of what the team's explored so far
> reduces the degradation of throughput as load increases, but still eventually collapses.
> Is that accurate? (I might have missed something the team's done that manages to converge
> to a stable point.)
> 
> RjS
> 
> 
>>>> I'm worried that we are taking yet another detour which will further delay the process.
>>> this would not delay the process in any way. If something, it could speed it up, because the requirements to review Experimental RFCs are lower than to review standards track RFCs... at the end of the day, I do not expect this to influence the time it takes to publish the specs because I am sure the group will make sure the quality of the specs as high as for a standards-track RFC.
>> I'd like to understand the concerns that lead to the proposal of putting this work through the experimental track.
>> 
>> What are we missing that prevents us from creating a standards track document?
>> 
>> Volker
>> 
>> 
>>

---

From: Volker Hilt <[email-masked]>
Date: Fri, 15 Jan 2010 10:57:37 -0500

Yes, we have the same results - the throughput stays at the capacity
limit and remains stable.

Volker

---

From: Janet P Gunn <[email-masked]>
Date: Fri, 15 Jan 2010 11:09:22 -0500

That was a concern with the very first work done by the SIP overload 
group.  But all the results presented recently have avoided that problem.

Janet

[email-masked] wrote on 01/15/2010 10:34:12 AM:

> [image removed] 
> Robert Sparks 
> to:
> Volker Hilt
> 01/15/2010 10:41 AM
> Sent by:
> [email-masked]
> 
> Cc:
> 
> DISPATCH, sip-overload, Gonzalo Camarillo, Lars Eggert
>

---

From: "Abdelal, Ahmed" <[email-masked]>
Date: Fri, 15 Jan 2010 17:04:26 -0500

We (Sonus) have implemented the SIP overload control and extensively
tested it. The results show that it mainatins the goodput under
different overload levels.

--Ahmed

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of NOEL, ERIC C
(ATTLABS)
Sent: Friday, January 15, 2010 10:57 AM
To: Volker Hilt; Henning Schulzrinne
Cc: DISPATCH; [email-masked]; Lars Eggert; Robert Sparks
Subject: Re: [sip-overload] [dispatch] Chartering SIP Overload

Same here at AT&T.

Eric

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Volker Hilt
Sent: Friday, January 15, 2010 10:58 AM
To: Henning Schulzrinne
Cc: DISPATCH; [email-masked]; Lars Eggert; Robert Sparks
Subject: Re: [sip-overload] [dispatch] Chartering SIP Overload

Volker

Henning Schulzrinne wrote:
> At least according to our measurements and simulations, that is not
the case. The throughput stays at ~100% until infinity.
team's explored so far
eventually collapses.
manages to converge
document?

---

From: "NOEL, ERIC C (ATTLABS)" <[email-masked]>
Date: Fri, 15 Jan 2010 10:56:44 -0500

Same here at AT&T.

Eric

-----Original Message-----

Volker

team's explored so far
eventually collapses.
manages to converge
document?

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Mon, 18 Jan 2010 15:04:32 +0100

I am worried that we are raising the bar on standards track documents.

I am certainly aware that some of the mechanisms described in the documents have been prototyped and have been shown to work. With that, the documentation merits standards track deliverables.

regards

Keith 

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Gonzalo Camarillo
> Sent: Friday, January 15, 2010 3:02 PM
> To: Hilt, Volker (Volker)
> Cc: DISPATCH; [email-masked]
> Subject: Re: [dispatch] Chartering SIP Overload
> 
> Hi,
> 
> this would not delay the process in any way. If something, it 
> could speed it up, because the requirements to review 
> Experimental RFCs are lower than to review standards track 
> RFCs... at the end of the day, I do not expect this to 
> influence the time it takes to publish the specs because I am 
> sure the group will make sure the quality of the specs as

---

From: Dean Willis <[email-masked]>
Date: Mon, 18 Jan 2010 12:08:47 -0600

On Jan 18, 2010, at 8:04 AM, DRAGE, Keith (Keith) wrote:

> I am certainly aware that some of the mechanisms described in the  
> documents have been prototyped and have been shown to work. With  
> that, the documentation merits standards track deliverables.

Judging from the number of "We have implemented this" responses, it  
might almost be ready to go from Proposed to Draft Standard.

--
Dean

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 18 Jan 2010 13:25:17 -0500

In addition, if multiple, independent simulations, published in peer-reviewed conferences, are no longer sufficient for standards-track designation, I'm at a bit of a loss to figure out what more any algorithm-containing protocol spec should do to avoid the "experimental" label. We constantly complain that we can't move specs up the maturity ladder; now, we can't even get *on* the ladder...

Henning

On Jan 18, 2010, at 1:08 PM, Dean Willis wrote:

> Judging from the number of "We have implemented this" responses, it might almost be ready to go from Proposed to Draft Standard.
> 
> --
> Dean

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 18 Jan 2010 12:56:32 -0600

Just following up here...

I sent a private e-mail to Robert, Lars and Magnus that might better have 
gone to the mailing list ...

My point was that

- I understand why TSV detours congestion avoidance drafts through 
Experimental, because we already have congestion avoidance mechanisms that 
actually work, so we should be cautious about dorking with them, but

- our standards-track SIP overload mechanisms have already been observed to 
fail on multiple live networks (I've heard this from two tier-one 
operators), so

- it seems appropriate to use a less restrictive strategy here - we 
shouldn't encourage people to ignore our maturity levels completely, and we 
shouldn't use our maturity levels to encourage people to continue deploying 
standards-track mechanisms that have already been observed to fail, and are 
not self-correcting when they fail, on production networks.

That fails the "first, do no harm" test. IMO, of course.

Spencer, whose first working group was a TSV-classic working group (PILC 
with Aaron Falk), and whose first RFC credit was on TCP congestion avoidance 
behavior ("TCP over Satellite", edited by Mark Allman)

>> documents have been prototyped and have been shown to work. With  that, 
>> the documentation merits standards track deliverables.
>
> Judging from the number of "We have implemented this" responses, it  might 
> almost be ready to go from Proposed to Draft Standard.
>
> --
> Dean

---

From: Lars Eggert <[email-masked]>
Date: Mon, 18 Jan 2010 21:17:50 +0200

Hi,

could someone send me a pointer to the latest experimental or simulation results for the various proposed mechanisms? I want to make sure I've seen the latest data.

I'm not on this list, so apologies if I have missed this.

> not self-correcting when they fail, on production networks.

I fully agree that the proposed mechanisms (that I've seen) are better than what we currently have (which is almost nothing.) I'm by no means opposed to doing this work. It's also extremely encouraging that these mechanisms are actively being developed, analyzed, deployed and monitored.

My main concern in a nutshell is: Do we have have one mechanism of which we are confident that it (1) operates very well in many deployment scenarios, (2) degrades gracefully (i.e., doesn't oscillate, etc.) in cases where it doesn't work very well and (3) is never worse than what we currently have?

If we do, then PS is appropriate. If we don't, then IMO publishing PS sends the wrong signal to deployers.

I've personally not seen enough data to convince me of (1) and (2) - but maybe I've missed some work, hence the request at the beginning of the email.

Lars

---

From: Volker Hilt <[email-masked]>
Date: Mon, 18 Jan 2010 15:06:19 -0500

Lars,

here is a list of papers on SIP overload that are referenced in 
http://tools.ietf.org/html/draft-ietf-sipping-overload-design-02:
- Hilt, V. and I. Widjaja, "Controlling Overload in Networks of SIP 
Servers", IEEE International Conference on Network Protocols (ICNP'08), 
Orlando, Florida, October 2008.
- Noel, E. and C. Johnson, "Initial Simulation Results That Analyze SIP 
Based VoIP Networks Under Overload", International Teletraffic Congress 
(ITC'07), Ottawa, Canada, June 2007.
- Shen, C., Schulzrinne, H., and E. Nahum, "Session Initiation Protocol 
(SIP) Server Overload Control: Design and Evaluation, Principles", 
Systems and Applications of IP Telecommunications (IPTComm'08), 
Heidelberg, Germany, July 2008.

More papers are referenced in these papers.

In addition, there were presentations at IETF meetings including
- IETF 74 SIPPING 
(http://www.ietf.org/proceedings/74/slides/sipping-11/sipping-11.htm),
- IETF 73 SIPPING 
(http://www.ietf.org/proceedings/73/slides/sipping-8/sipping-8.htm),
- IETF 71 TSVAREA 
(http://www.ietf.org/proceedings/71/slides/tsvarea-3/tsvarea-3.htm)

Volker

On 1/18/2010 2:17 PM, Lars Eggert wrote:
> Hi,
>
> could someone send me a pointer to the latest experimental or
> simulation results for the various proposed mechanisms? I want to
> make sure I've seen the latest data.
>
>> - it seems appropriate to use a less restrictive strategy here -
>> we shouldn't encourage people to ignore our maturity levels
>> completely, and we shouldn't use our maturity levels to encourage
>> people to continue deploying standards-track mechanisms that have
>> already been observed to fail, and are not self-correcting when
>> they fail, on production networks.
>
> I fully agree that the proposed mechanisms (that I've seen) are
> better than what we currently have (which is almost nothing.) I'm by
> no means opposed to doing this work. It's also extremely encouraging
> that these mechanisms are actively being developed, analyzed,
> deployed and monitored.
>
> My main concern in a nutshell is: Do we have have one mechanism of
> which we are confident that it (1) operates very well in many
> deployment scenarios, (2) degrades gracefully (i.e., doesn't
> oscillate, etc.) in cases where it doesn't work very well and (3) is
> never worse than what we currently have?
>
> I've personally not seen enough data to convince me of (1) and (2) -
> but maybe I've missed some work, hence the request at the beginning
> of the email.
>
> Lars

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 18 Jan 2010 17:43:04 -0500

> My main concern in a nutshell is: Do we have have one mechanism of which we are confident that it (1) operates very well in many deployment scenarios, (2) degrades gracefully (i.e., doesn't oscilla…

I don't see how you can prove a negative, even if we simulate (or implement) for another ten years. There's always the theoretical possibility that some constellation of the moon and the operating system will cause undesirable behavior. Thus, your bar is provably impossible to meet.

That said, I think the proponents of various congestion control mechanisms can indeed do a better job describing the inherent limitation of some of the algorithms. In particular, the issue is with cases of a very large number of upstream proxies (say, hundreds to thousands). Just as with flow control, you can design for inherent safety, but there's a limit of how many active connections you can allow. In a UDP-based system where all the world's proxies can theoretically conspire to send a SIP INVITE to the "victim" at 5 pm EST today, no SIP congestion control can help since the "victim" can't know that universe of senders. But TCP (or SCTP) has exactly the same problem, at connection setup. This is one instance of the SIP avalanche problem discussed separately, but which hasn't received as much attention.

Thus, I think the problem is more usefully scoped, we might make more progress and limit what needs to be evaluated.

> I've personally not seen enough data to convince me of (1) and (2) - but maybe I've missed some work, hence the request at the beginning of the email.

It would be more productive, I think, if there was a concrete set of benchmarks or test cases. In general, not reflecting your particular remarks, I find the typical review remark of "the authors of paper X should do more measurements" rather unhelpful.

In response to another email: The analogy to congestion control is only helpful to a limited extent, in my view. The system (and algorithms) are much closer to TCP flow control, since you don't have the third party effects ("cross traffic") and the primary limitation is avoiding buffer overflow (or, rather, SIP timer triggering, which amounts to roughly the same thing).

Henning

> 
> Lars_______________________________________________
> dispatch mailing list
> [email-masked]