Subject: Re: [ledbat] [R-C] LEDBAT vs RTCWeb

From: Randell Jesup <[email-masked]>
Date: Fri, 13 Apr 2012 05:17:31 -0400

On 4/13/2012 3:03 AM, Stefan Holmer wrote:
>
>
> On Thu, Apr 12, 2012 at 10:52 PM, Randell Jesup <[email-masked]
> <mailto:[email-masked]>> wrote:

>     It's *possible* that RRTCC will 'win', it's also definitely possible
>     that they'll end up semi-stable where neither goes too far away from
>     a 'fair' share.  The one thing I don't predict is stable, fair and
>     predictable sharing of the bandwidth.  :-/
>
>
> I agree. Because of different averaging and trigger thresholds they will
> likely not end up in a stable state. It for sure seems unlikely that
> they will happen to fairly share the bandwidth.

The real problem here is that LEDBAT is designated a "scavenger" 
protocol that should get out of the way of primary uses (which includes 
rtcweb traffic).  While it's possible that will be the result 
experimentally, I tend to doubt it and it's certainly unclear without 
experiments - and I also doubt a fair sharing will occur.  So my guess 
(which should be checked!) is that LEDBAT and RRTCC are not compatible 
on the same bottleneck links.  This means that manual intervention will 
be needed to enable RRTCC traffic to be usable; either stopping or 
bandwidth-limiting any LEDBAT flows.

In theory if the OS was controlling the LEDBAT flows it could be asked 
by an RRTCC (userspace) application to have them get out of the way 
(which probably means halt or virtually so during RRTCC operation) or to 
in some manner use send() traffic as a flag to do so.  An example might 
be applications using LEDBAT in OSX for 'background' download/update 
that may not have external controls that a user could use to suspend 
transfers during a call.  I'm not holding my breath on this one; and it 
wouldn't help if there's another endpoint behind the same bottleneck 
using LEDBAT.

The last recourse is the advanced modem/router with classification 
(again, not something we can do anything about).  However, as Jim Gettys 
will tell you, this may not help you as much if another link is the 
bottleneck, such as a wifi router behind the modem or primary router.

I think we're going to find that LEDBAT has failed in (what should be) a 
primary goal, which is to avoid hurting "foreground" traffic, largely 
because they appear to have only considered TCP flows (from review of 
their mailing list and specs).  Regular inflexible VoIP traffic is 
likely badly hurt by the current spec (since 100+ms of extra delay will 
push typical VoIP traffic well into the "bad" part of the MOS slope), 
and delay-sensing foreground protocols like RRTCC are probably blown out 
of the water.

If LEDBAT actually is to be a 'scavenger' protocol, it must have some 
mechanism other than purely queue depth to allow foreground protocols to 
push it out of the way.  It's possible it could stick to queue depth but 
use very small values, and/or use slower time constants than 
"foreground" delay-sensing algorithms to guarantee they 'win' when 
competing with it.

Cross-posting to the LEDBAT list in the hopes that I'm wrong.  (For 
reference, RRTCC is a delay-sensing CC algorithm for RTP traffic, 
recently discussed at IETF83 in the ICCRG and planned for use in rtcweb
clients.  RRTCC is a brand-new moniker for the algorithm in Harald 
Alvestrand's draft, but similar algorithms have been in use (but not 
standardized) since at least 2004, long predating LEDBAT/uTP.)

-- 
Randell Jesup
[email-masked]

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 20 Apr 2012 13:55:36 +0200

Hi Randell,

I didn't follow the whole discussion but regarding LEDBAT we have a TARGET 
delay of max. 100ms. That means you can choose a smaller one. We've chosen 
100ms as a max as there is an ITU recommendation that 150 ms delay is 
acceptable for most user voice applications and we wanted for sure stay below 
that.

If you choose a delay-based congestion control I don't think your problem is 
LEDBAT but standard loss-based TCP that will frequently fill up the queue 
completely. 

Maybe you don't want to look at the total queuing delay but at the changes in 
queuing delay...? LEDBAT will keep the delay constant.

Mirja

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Michael Welzl <[email-masked]>
Date: Fri, 20 Apr 2012 14:32:42 +0200

On Apr 20, 2012, at 2:23 PM, Jim Gettys wrote:

> On 04/20/2012 07:55 AM, Mirja Kuehlewind wrote:
>> Hi Randell,
>>
> 100 ms + 75ms speed of light delay across the US (or equivalent across
> Europe, for example) + 100ms at the receiving end....
>
> Of course, it's even worse between continents, even without broken  
> networks.
>
> Not so nice....

Not argueing about your point here (I agree that we have to fix the  
edge), but: LEDBAT is an end-to-end mechanism, so I think that the  
100ms reflect the total measured end-to-end delay.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Fri, 20 Apr 2012 14:51:18 +0200

On Apr 20, 2012, at 2:39 PM, Stefan Holmer wrote:

>
>
> On Fri, Apr 20, 2012 at 2:32 PM, Michael Welzl <[email-masked]>  
> wrote:
>
> Is this really the case? I interpret that the target (100 ms) refers  
> to queueing delay, since LEDBAT tries to minimize target -  
> queueing_delay, where queueing_delay = current_delay - base_delay.  
> Could be wrong though.

Sorry, my bad (I think).

Cheers,
Michael

---

From: Randell Jesup <[email-masked]>
Date: Fri, 20 Apr 2012 10:48:15 -0400

> acceptable for most user voice applications and we wanted for sure stay below

I'm afraid you've mis-understood the 150ms number.  That's the "knee" in 
the curve for mouth-to-ear delay, not congestion delay or even 
end-to-end packet delay.  (And it's more complicated than that; the 
150ms number is dependent on the amount of echo from the far end - with 
high echo (poor/no ECs), it can be smaller.)  And you can get asymmetric 
affects where delays in each direction are unequal.

For VoIP communication, the delay budget roughly looks like this:

frame size    - typ 20-30ms -- you have to buffer up one packet to send
echo cancellation - typ 1-3ms?
encoder delay - typ 1-2ms?
algorithmic delay - typ 0-5ms
packetization, output queuing, etc - 0-10ms (typically low/nil)
unloaded (no local-loop queuing) transit time: typically 20-100ms)
queuing delay: ?
jitter buffer depth - typically 20-60ms
decoder, time scale modifier (adaptive jitter buffer): 0-2ms?
rebuffering into audio frames for drivers: typ 1/2 frame size (5-10ms)
Other random signal processing: 0-2ms?
output device driver buffering (and reframing in OS frame size chunks - 
16ms typ on Linux for 8KHz audio) - typ 10ms?  longer on some OS's!!!
hardware buffers

This is kind of abstract and people could argue the numbers or list, but 
it's to give you an idea that queue depth is far from the only item 
(though it's the most variable one).  Almost all of these are fixed 
and/or small, except transit time, queuing delay and jitter buffer depth 
(indirectly affected by queuing).

Take off the fixed/small items from 150ms, and you are probably left 
80-100ms (if you're lucky, 50-80 if you're not) - for transit, jitter 
buffer and queuing (and video calls can be a bit worse, with longer 
frame lengths, more jitter and often longer hardware queues).  So, to 
stay under 150ms on local hops (with a fast access link at both ends), 
you need moderate jitter and can probably handle some static queuing 
(<25-50ms).  For longer routes and/or slower access links (DSL), there's 
basically no budget for standing queues, especially as jitter is 
typically higher.

I guarantee you any VoIP engineer seeing "100ms queuing delay" has their 
heart sink about conversational quality.  Yes, you can have calls.  Yes, 
they *will* suffer "typical" awkward-pause/talkover.  You'll probably 
generally end up in the 200-300ms range mouth-ear, which isn't at the 
~400-500ms "What do you think? over!" walkie-talkie level, but is 
uncomfortable.

And that's assuming old-style inflexible VoIP UDP streams (G.711, G.722, 
G.729 (ugh).  Once you add video with BW adaptation or adaptive audio 
codecs, interacting with LEDBAT gets painful if the VoIP stream uses a 
delay-sensing protocol (and it really, really wants to).

> LEDBAT but standard loss-based TCP that will frequently fill up the queue

Delay-based can't "beat" a loss-based TCP flow that's long enough, 
that's true, but luckily most TCP flows are relatively short and/or 
bursty, especially across the access link.

> queuing delay...? LEDBAT will keep the delay constant.

RRTCC and similar algorithms do not use OWD estimates, and so are less 
sensitive to mis-measurement of the base delay (which from the LEDBAT 
simulations can cause problems).  RRTCC works entirely from deltas in 
inter-packet delay to determine if the queue is growing or shrinking (or 
stable).  After a queuing event is observed (growing queue enough to 
give a signal from the filter), it drops bandwidth and tries to stay 
down (not probe for extra bandwidth) until the queue has drained and is 
once again stable.  This allows it to generally make close to full use 
of bandwidth available with close to 0-length queues.  It does generally 
value low queues over 100% bandwidth efficiency.

-- 
Randell Jesup
[email-masked]

---

From: Rolf Winter <[email-masked]>
Date: Mon, 23 Apr 2012 12:07:25 +0000

Hi,

the way I remember how we got to the 100ms was a little different. We started out with 25 ms of extra delay and between version 01 and 02 of the WG draft I believe it "magically" changed. I remember asking about the motivation but I cannot recall an answer. My guess is that 25ms is very few packets in common residential broadband access networks and the algorithm might not quite work that well in those settings _today_. After some debate we ended up accepting 100 ms as a maximum. The problem is that going away from 100 ms in practice is that (assuming more than one app adopts LEDBAT congestion control) the one app that yields the most will get the least throughput, making anything less than 100 ms really unattractive in a competitive environment. So while I am personally OK with the compromise we achieved in the end, I am not happy about the lack of motivation. I am mildly optimistic that it is based on experience in the field rather than making the existing implementation at the time match the spec.

Best,

Rolf

NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Randell Jesup
> Sent: Freitag, 20. April 2012 16:48
> To: [email-masked]; [email-masked]
> Subject: Re: [ledbat] [R-C] LEDBAT vs RTCWeb
> 
> > I didn't follow the whole discussion but regarding LEDBAT we have a
> > TARGET delay of max. 100ms. That means you can choose a smaller one.
> > We've chosen 100ms as a max as there is an ITU recommendation that
> 150
> > ms delay is acceptable for most user voice applications and we wanted
> > for sure stay below that.
> 
> I'm afraid you've mis-understood the 150ms number.  That's the "knee"
> in the curve for mouth-to-ear delay, not congestion delay or even end-
> to-end packet delay.  (And it's more complicated than that; the 150ms
> number is dependent on the amount of echo from the far end - with high
> echo (poor/no ECs), it can be smaller.)  And you can get asymmetric
> This is kind of abstract and people could argue the numbers or list,
> but it's to give you an idea that queue depth is far from the only item
> and/or small, except transit time, queuing delay and jitter buffer
> depth (indirectly affected by queuing).
> 
> (<25-50ms).  For longer routes and/or slower access links (DSL),
> there's basically no budget for standing queues, especially as jitter
> is typically higher.
> 
> I guarantee you any VoIP engineer seeing "100ms queuing delay" has
> their heart sink about conversational quality.  Yes, you can have calls.
> Yes, they *will* suffer "typical" awkward-pause/talkover.  You'll
> probably generally end up in the 200-300ms range mouth-ear, which isn't
> at the ~400-500ms "What do you think? over!" walkie-talkie level, but
> is uncomfortable.
> 
> > If you choose a delay-based congestion control I don't think your
> > problem is LEDBAT but standard loss-based TCP that will frequently
> > fill up the queue completely.
> 
> > Maybe you don't want to look at the total queuing delay but at the
> > changes in queuing delay...? LEDBAT will keep the delay constant.
> 
> inter-packet delay to determine if the queue is growing or shrinking
> (or stable).  After a queuing event is observed (growing queue enough
> to give a signal from the filter), it drops bandwidth and tries to stay
> of bandwidth available with close to 0-length queues.  It does
> generally value low queues over 100% bandwidth efficiency.
> 
> 
> --
> Randell Jesup
> [email-masked]
> _______________________________________________
> ledbat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ledbat

---

From: Randell Jesup <[email-masked]>
Date: Mon, 23 Apr 2012 12:57:58 -0400

On 4/23/2012 8:07 AM, Rolf Winter wrote:
> Hi,
>
> the way I remember how we got to the 100ms was a little different. We started out with 25 ms of extra delay and between version 01 and 02 of the WG draft I believe it "magically" changed. I remember asking about the motivation but I cannot recall an answer. My guess is that 25ms is very few packets in common residential broadband access networks and the algorithm might not quite work that well in those settings _today_. After some debate we ended up accepting 100 ms as a maximum. The problem is that going away from 100 ms in practice is that (assuming more than one app adopts LEDBAT congestion control) the one app that yields the most will get the least throughput, making anything less than 100 ms really unattractive in a competitive environment. So while I am personally OK with the compromise we achieved in the end, I am not happy about the lack of motivation. I am mildly optimistic that it is based on experience in the field rather than making the existing implementation at the tim
>     atch the spec.

I searched the email archives and didn't see anything discussing it 
(though I might have missed it) or discussing interaction with VoIP 
(other than that one paper, which also showed the sensitivity to the 
allowed delay and the advantage of a user willing to accept longer 
delays, plus how latecomers could mis-read the buffer state.

The problem is that a 100ms-delay-scavenger algorithm like this "poisons 
the well" for any application that desires/needs lower delay.  That's 
why I believe there are only two reasonable targets: 0 (drained queues) 
and full queues (assuming AQM isn't implemented to keep queues low).

I don't think this is just a theoretical problem, I think it's a very 
serious issue, which if LEDBAT gets used a lot will cause a lot of 
problems.  Perhaps it's *better* than old-style saturating bittorrent 
flows, in that VoIP will be possible without shutting down bittorrent, 
but that doesn't mean it's good, and the perception that bittorrent 
doesn't break other apps with the new protocols will encourage people to 
leave it running - and depending on the vagaries of bittorrent, it might 
be fine when you start the call and then slam you.  I'm especially 
worried about OS's and apps using it for background update transfers 
with no user control or knowledge, etc.

-- 
Randell Jesup
[email-masked]

---

From: "Woundy, Richard" <[email-masked]>
Date: Mon, 23 Apr 2012 19:37:39 +0000

>> the way I remember how we got to the 100ms was a little different. We started out with 25 ms of extra delay and between version 01 and 02 of the WG draft I believe it "magically" changed. I remember asking about the motivation but I cannot recall an answer. My guess is that 25ms is very few packets in common residential broadband access networks and the algorithm might not quite work that well in those settings _today_. After some debate we ended up accepting 100 ms as a maximum. 

I seem to remember a very simple motivation: use the same delay target (100 ms) that was used in the BitTorrent uTP implementation, since uTP has seen significant production network deployment.

-- Rich
________________________________________
From: [email-masked] [[email-masked]] on behalf of Randell Jesup [[email-masked]]
Sent: Monday, April 23, 2012 12:57 PM
To: Rolf Winter
Cc: [email-masked]; [email-masked]
Subject: Re: [ledbat] [R-C]   LEDBAT vs RTCWeb

--
Randell Jesup
[email-masked]

---

From: Rolf Winter <[email-masked]>
Date: Mon, 23 Apr 2012 21:27:48 +0000

> > the way I remember how we got to the 100ms was a little different. We
> started out with 25 ms of extra delay and between version 01 and 02 of
> the WG draft I believe it "magically" changed. I remember asking about
> the motivation but I cannot recall an answer. My guess is that 25ms is
> very few packets in common residential broadband access networks and
> the algorithm might not quite work that well in those settings _today_.
> After some debate we ended up accepting 100 ms as a maximum. The
> problem is that going away from 100 ms in practice is that (assuming
> more than one app adopts LEDBAT congestion control) the one app that
> yields the most will get the least throughput, making anything less
> than 100 ms really unattractive in a competitive environment. So while
> I am personally OK with the compromise we achieved in the end, I am not
> happy about the lack of motivation. I am mildly optimistic that it is
> based on experience in the field rather than making the existing
> implementation at the tim
> >     atch the spec.
> 
> The problem is that a 100ms-delay-scavenger algorithm like this
> "poisons the well" for any application that desires/needs lower delay.
> That's why I believe there are only two reasonable targets: 0 (drained
> queues) and full queues (assuming AQM isn't implemented to keep queues
> low).

I think worrying about a poisoned well is a little late with loss-based congestion control being deployed.
 
> doesn't break other apps with the new protocols will encourage people
> to leave it running - and depending on the vagaries of bittorrent, it
> might be fine when you start the call and then slam you.  I'm
> especially worried about OS's and apps using it for background update
> transfers with no user control or knowledge, etc.

Well, this reads a bit too dramatic for my taste. LEDBAT typically comes with the app so you're a bit more agile regarding the congestion control parameterization (and even the implementation as the BitTorrent folks have shown). BTW, as long as update services do not use the uplink capacity of customers, I don't see much of a problem (assuming we agree that the biggest problem is home gateways). Not sure how BITS compares in all of this either. In either case it beats TCP and that is used without the users' knowledge, too. The best you could probably do is design better home gateways which would solve many of the problems we see today (the better ones actually today implement traffic shaping with really good results, probably better than any end-to-end congestion control algorithm).

Best, 	

Rolf

---

From: Randell Jesup <[email-masked]>
Date: Mon, 23 Apr 2012 17:56:43 -0400

> I think worrying about a poisoned well is a little late with loss-based congestion control being deployed.

We know when faced with deep buffers and large loss-based flow, a delay 
flow will lose or have to transition to loss-based and live with delay 
(see Cx-TCP).  We also know that in practice, the (residential) 
bottleneck links are almost always the access link (DSL, fiber, etc) or 
sometimes the WiFi connection. Corporate/educational can be a bit 
different, but there's a better chance of AQM or service-based traffic 
shaping there.

We can deal (mostly) with fighting with TCP and expected it.  We didn't 
expect scavenger protocols to be pumping the queues up when TCP is idle 
or bursty.

> Well, this reads a bit too dramatic for my taste.

Perhaps.  But I think a fundamental "how does this affect the network" 
case was glossed over/ignored.

>   LEDBAT typically comes with the app so you're a bit more agile regarding the congestion control parameterization (and even the implementation as the BitTorrent folks have shown).

I note that LEDBAT is now in Linux and MacOS, so it's not just in 
application code anymore, and app developers (even OS developers) may 
assume it's safe to blindly use without user involvement.  (And even if 
informed, assuming users understand the interaction of protocols is ... 
a stretch.)

> BTW, as long as update services do not use the uplink capacity of customers, I don't see much of a problem (assuming we agree that the biggest problem is home gateways).

It's not just uplink, but also downlinks (though uplinks typically are 
the worst-configured).  An update service for the OS or a large app can 
fill the downstream buffers with 100ms of packets, and then any incoming 
VoIP data is delayed with no real way to solve it until the 
update/download ends.  In some cases that might be hours or even days.  
Other apps like cloud backup services might assume it's safe to use 
LEDBAT for their background transfers, and keep your upstream saturated 
at 100ms delay at all times.

> Not sure how BITS compares in all of this either. In either case it beats TCP and that is used without the users' knowledge, too. The best you could probably do is design better home gateways which would solve many of the problems we see today (the better ones actually today implement traffic shaping with really good results, probably better than any end-to-end congestion control algorithm).

Note that no routers are likely to recognize rtcweb traffic as VoIP 
since the signaling is opaque and application-dependant, so ALGs and the 
like have nothing to work against.  It would have to 'guess' that the 
packet stream leading byte pattern and STUN/ICE traffic signified use of 
the port for VoIP.  Eventually they may learn, but it will take years.

I stand by the contention this is a real, significant problem which 
could get far worse if non-user-centric services start using LEDBAT.

-- 
Randell Jesup
[email-masked]

---

From: Rolf Winter <[email-masked]>
Date: Tue, 24 Apr 2012 20:35:23 +0000

> >> queues) and full queues (assuming AQM isn't implemented to keep
> >> queues low).

Here is where you lose me a bit. How do you "fight" TCP and its queue build-up exactly. I don't think we talk about protocols with the same design goals, are we. One of LEDBAT's design goals e.g. is to use all available bandwidth.

> >> flows, in that VoIP will be possible without shutting down
> >> bittorrent, but that doesn't mean it's good, and the perception that
> >> bittorrent doesn't break other apps with the new protocols will
> >> encourage people to leave it running - and depending on the vagaries
> >> of bittorrent, it might be fine when you start the call and then
> slam
> >> you.  I'm especially worried about OS's and apps using it for
> >> background update transfers with no user control or knowledge, etc.
> >   LEDBAT typically comes with the app so you're a bit more agile
> regarding the congestion control parameterization (and even the
> implementation as the BitTorrent folks have shown).
> 

And I think it is good that LEDBAT sees some deployment. It is an experimental congestion control algorithm. These experiments will provide us with good feedback and we might need to revisit some design decisions. There are millions of deployed LEDBAT-enabled clients out there and unfortunately the only thing we know about this "experiment" is that the experimenters have assured us that LEDBAT was deployed with "good success". 

> 
> > BTW, as long as update services do not use the uplink capacity of
> customers, I don't see much of a problem (assuming we agree that the
> biggest problem is home gateways).
> 
> fill the downstream buffers with 100ms of packets, and then any
> incoming VoIP data is delayed with no real way to solve it until the
> > Not sure how BITS compares in all of this either. In either case it
> beats TCP and that is used without the users' knowledge, too. The best
> you could probably do is design better home gateways which would solve
> many of the problems we see today (the better ones actually today
> implement traffic shaping with really good results, probably better
> than any end-to-end congestion control algorithm).
> 
> since the signaling is opaque and application-dependant, so ALGs and
> the like have nothing to work against.  It would have to 'guess' that
> the packet stream leading byte pattern and STUN/ICE traffic signified
> use of the port for VoIP.  Eventually they may learn, but it will take
> years.
> 

Maybe, but the alternative protocol that these apps you describe above can use is TCP. I remain unconvinced that TCP makes the problem less severe than LEDBAT would.

---

From: Randell Jesup <[email-masked]>
Date: Tue, 24 Apr 2012 18:01:57 -0400

> Here is where you lose me a bit. How do you "fight" TCP and its queue build-up exactly. I don't think we talk about protocols with the same design goals, are we. One of LEDBAT's design goals e.g. i…

Our (WebRTC/RRTCC's) primary constraints are to a) keep delay near 0, b) 
use a 'fair' share of bandwidth (all available if the other flows don't 
use all the available bandwidth).  Low delay generally takes precedence 
over bandwidth, though eventually we hit a wall and can't reasonably 
reduce bandwidth more.

The design goals are not identical to LEDBAT; but these protocols need 
to share the network "well" with each other and TCP (and inflexible UDP 
flows like classic VoIP).  TCP is a given (unfortunately), and drop-tail 
routers with deep buffers are also a given (very unfortunately, 
especially with TCP).

We can't stop TCP from grabbing all buffers; though there may be limited 
ability to "knock TCP back" some periodically when there are a small 
number of competing TCP flows, at the costs of delay bursts - but that 
probably isn't practical (a usable experience) for users.

> And I think it is good that LEDBAT sees some deployment. It is an experimental congestion control algorithm. These experiments will provide us with good feedback and we might need to revisit some d…

Millions != experiments.  This is wide-scale deployment, and implying it 
isn't is misleading (as will deployment of WebRTC).  And implementation 
in core OS/X and Linux one assumes will lead to use by OS functions and 
applications such as cloud file backup, app update, etc.

So if this "breaks" the network, we have a problem.

> Maybe, but the alternative protocol that these apps you describe above can use is TCP. I remain unconvinced that TCP makes the problem less severe than LEDBAT would.

I think this is a false dichotomy, in that there are other choices for 
such applications, such as TFRC, RRTCC (which we're defining), 
TCP-Vegas(?), Cx-TCP (which in testing was ~20ms delay, but switches to 
loss-based fair sharing when faced with TCP - modifying it can probably 
lead to it dropping to low BW when competing) and for that matter LEDBAT 
itself (or a variant thereof) with different tunings.

I believe RRTCC could cover a fair amount of of the goals of LEDBAT with 
alternate tunings, for example.

Let me be clear: *if* widespread LEDBAT use, especially by 
"background"/non-user-visible apps will cause major disruption of VoIP 
flows on the net (and in particular for flows from within the same 
residence or enterprise), then there's a real problem with encouraging 
deployment.  If it does not, then the issue is minor.  I also believe 
that we do not know the answer to this question currently, though I 
believe the answer it is that it will cause significant delay and 
quality loss, and we have some evidence to that effect.

Also (and I haven't checked this yet): how does LEDBAT respond to AQM?  
 From telecom-paristech.fr:

    The experimental work in [ITC22] exploits instead a own
    implementation and points that LEDBAT may not work as expected (and
    actually might be needed) in case active queue management techniques
    AQM are applied in user modems (the study also suggests AQM to
    become a default practice in set-top-box devices)

http://www.infres.enst.fr/~drossi/DATA/PRJ-Ledbat+AQM.pdf

It seems like AQM causes LEDBAT to promote to behavior similar to TCP, 
but with low delay.  Since it still seems fair, this is ok, but it's no 
longer a background or scavenger flow, and applications using it need to 
be aware they can impact "foreground" flows if AQM is in play.  Perhaps 
applications need to be given awareness when LEDBAT detects active AQM 
(if it can), and there's no "scavenger" CC algorithm for AQM that I know of.

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Tue, 24 Apr 2012 20:08:29 -0400

On 4/24/2012 7:23 PM, Bill Ver Steeg (versteb) wrote:
>
> I am a bit confused by the last paragraph. I understand why one would 
> implement active queue management algorithms in middleboxes like 
> cable/DSL modems, but I am not sure how AQM on a set top box would 
> help with bufferbloat. I guess I could come up with some very narrow 
> cases in which an endpoint would benefit from advance buffer 
> management techniques for self generated competing flows, but this 
> would be very artificial and of no use in the real world.
>

That's a quote from telecom-paristech.fr; it may be a mis-translation, 
or they may be referring to cable modems as "set top" devices.

-- 
Randell Jesup
[email-masked]

---

From: Rolf Winter <[email-masked]>
Date: Thu, 26 Apr 2012 15:05:55 +0000

> 	Here is where you lose me a bit. How do you "fight" TCP and its
> queue build-up exactly. I don't think we talk about protocols with the
> same design goals, are we. One of LEDBAT's design goals e.g. is to use
> all available bandwidth.
> 
> 
> Our (WebRTC/RRTCC's) primary constraints are to a) keep delay near 0,
> b) use a 'fair' share of bandwidth (all available if the other flows
> don't use all the available bandwidth).  Low delay generally takes
> precedence over bandwidth, though eventually we hit a wall and can't
> reasonably reduce bandwidth more.
> 
> flows like classic VoIP).  TCP is a given (unfortunately), and drop-
> tail routers with deep buffers are also a given (very unfortunately,
> especially with TCP).
> 
> We can't stop TCP from grabbing all buffers; though there may be
> limited ability to "knock TCP back" some periodically when there are a
> small number of competing TCP flows, at the costs of delay bursts - but
> that probably isn't practical (a usable experience) for users.

Exactly. You cannot. Not sure what's behind "knock TCP back" but as you say you'll add delay which pretty much seems to violate your own design goals. It generally becomes difficult for me to discuss something that I haven't seen a spec of. Is there a document that outlines how this congestion control algorithm actually works? 

> 	And I think it is good that LEDBAT sees some deployment. It is an
> experimental congestion control algorithm. These experiments will
> provide us with good feedback and we might need to revisit some design
> decisions. There are millions of deployed LEDBAT-enabled clients out
> there and unfortunately the only thing we know about this "experiment"
> is that the experimenters have assured us that LEDBAT was deployed with
> "good success".
> 
> 
> Millions != experiments.  This is wide-scale deployment, and implying
> it isn't is misleading (as will deployment of WebRTC).  And
> implementation in core OS/X and Linux one assumes will lead to use by
> OS functions and applications such as cloud file backup, app update,
> etc.
> 

As far as the IETF goes the congestion control spec is experimental. I also think the spec does a decent job when it comes to underlying assumptions made such as FIFO queues. Also it recommends to use AQM and acknowledges the fact that its behavior will look much more like TCP when AQM is employed etc. The way this work came to the IETF was obviously after the fact, but in terms of congestion control this is not without precedent. That doesn't mean the IETF should elevate every widely deployed technology to proposed standard but having it documented, having expert reviews and encouraging experiments is a good thing. And that these "things" can and do happen is just due to the way the Internet was built. When you build your congestion control algorithm you shouldn't expect that you'll only have to deal with TCP and that it will actually behave the way it does today for eternity.

> 	Maybe, but the alternative protocol that these apps you describe
> above can use is TCP. I remain unconvinced that TCP makes the problem
> less severe than LEDBAT would.
> 
> 
> such applications, such as TFRC, RRTCC (which we're defining), TCP-
> Vegas(?), Cx-TCP (which in testing was ~20ms delay, but switches to
> lead to it dropping to low BW when competing) and for that matter
> LEDBAT itself (or a variant thereof) with different tunings.

I think we disagree here. I take that you're defining RRTCC just now so a couple of years back that wasn't available. When you talk about any TCP-flavor and not just about the congestion control part you are correct in theory, however in practice you're not since you'll need to wait till all OS folks have implemented it. Clearly that is not an option if you want to roll out and sell your app today.
 
> I believe RRTCC could cover a fair amount of of the goals of LEDBAT
> with alternate tunings, for example.

A pointer to the spec would be highly appreciated.

> we have some evidence to that effect.

Data? A pointer would be nice.

> AQM are applied in user modems (the study also suggests AQM to become a
> default practice in set-top-box devices)

I love it when people point to papers that I have co-authored ;) We haven't looked at set-top-boxes but home gateways. And indeed the ones that cost a little more come with useable default settings for traffic shaping where the combo TCP BitTorrent plus Voice beats LEDBAT plus Voice (by miles). But as you stated earlier, this relies on the fact that the box knows how to tell them apart. The downside is that they don't do congestion control obviously. So if you intend to send lot's of data you'll still need to do that.

> longer a background or scavenger flow, and applications using it need
> to be aware they can impact "foreground" flows if AQM is in play.
> Perhaps applications need to be given awareness when LEDBAT detects
> active AQM (if it can), and there's no "scavenger" CC algorithm for AQM
> that I know of.

And the document makes that clear and I am not sure LEDBAT actually can detect AQM.

---

From: Nicholas Weaver <[email-masked]>
Date: Thu, 26 Apr 2012 08:49:18 -0700

> And the document makes that clear and I am not sure LEDBAT actually can detect AQM.

One thought:  Active Queue management will be either ECN or early drop.  Which is a signal separate to the delay signal used which is "friendlier than TCP".

In either case, the scavenger flow property might be, well, not fully maintained but at least encouraged by backing off more than conventional TCP would to the same signal.

---

From: Randell Jesup <[email-masked]>
Date: Thu, 26 Apr 2012 13:12:52 -0400

> In either case, the scavenger flow property might be, well, not fully maintained but at least encouraged by backing off more than conventional TCP would to the same signal.

Correct, and that's a reasonable way to proceed - though it might 
complicate slightly (and certainly change) the LEDBAT competing with 
LEDBAT case.

In my delay-sensitive CC algorithm, I detected drops, and in particular 
"fishy" drops where the end-to-end delay in the following packet was 
lower by roughly the normal arrival interval, and would take those as an 
indication that we should drop more substantially than 'normal'.  This 
was targeted at tail-drop detection, especially congestion spikes, but 
for a scavenger protocol the same idea could apply to make it more 
responsive to AQM.

Basically, for a scavenger protocol any drop is a warning that either 
you or someone else has pushed a queue up to tail-drop levels, or that 
AQM is in play.  In either case you should back off, and if you back off 
more than TCP would, that that allows TCP to retain it's foreground 
advantage while not hurting scavenger performance (much?) when it 
competes with itself.  It may want to use drops to develop an estimate 
of the bottleneck queue depth (if less than the target value), to help 
avoid triggering tail-drops when not competing with TCP.  (I.e. a target 
of 100ms doesn't make sense if the bottleneck queue is 50ms deep - and 
you can consider AQM really the equivalent of a minimal-depth queue at 
the bottleneck.)  Such an estimate (which would be near 0 for AQM) might 
let it properly handle AQM, which is an idea which might help in RRTCC 
as well.

For those looking for the "RRTCC" (horrible name :-)  draft, here's the 
latest:

http://tools.ietf.org/html/draft-alvestrand-rtcweb-congestion-02

From: [email-masked]

This is mainly a maintenance update. No changes to the algorithm, but 
reworded to show how it can be applied across multiple SSRCs at one time.

                  Harald

-------- Original Message --------
Subject: 	New Version Notification for 
draft-alvestrand-rtcweb-congestion-02.txt
Date: 	Wed, 25 Apr 2012 05:03:35 -0700
From: 	[email-masked]
To: 	[email-masked]
CC: 	[email-masked]

A new version of I-D, draft-alvestrand-rtcweb-congestion-02.txt has been successfully submitted by Harald Alvestrand and posted to the IETF repository.

Filename:	 draft-alvestrand-rtcweb-congestion
Revision:	 02
Title:		 A Google Congestion Control Algorithm for Real-Time Communication on the World Wide Web
Creation date:	 2012-04-25
WG ID:		 Individual Submission
Number of pages: 17

Abstract:
    This document describes two methods of congestion control when using
    real-time communications on the World Wide Web (RTCWEB); one sender-
    based and one receiver-based.

    It is published to aid the discussion on mandatory-to-implement flow
    control for RTCWEB applications; initial discussion is expected in
    the RTCWEB WG&#39;s mailing list.

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Fri, 27 Apr 2012 14:45:43 -0400

On 4/27/2012 7:48 AM, Stefan Holmer wrote:
>
>
> On Thu, Apr 26, 2012 at 7:12 PM, Randell Jesup <[email-masked]
>>>>     to be aware they can impact"foreground"  flows if AQM is in play.
>>>>     active AQM (if it can), and there's no"scavenger"  CC algorithm for AQM
>>>>     that I know of.
>>     One thought:  Active Queue management will be either ECN or early drop.  Which is a signal separate to the delay signal used which is"friendlier than TCP".
>>
>     In my delay-sensitive CC algorithm, I detected drops, and in
>     particular "fishy" drops where the end-to-end delay in the following
>     packet was lower by roughly the normal arrival interval, and would
>     take those as an indication that we should drop more substantially
>     than 'normal'.  This was targeted at tail-drop detection, especially
>     congestion spikes, but for a scavenger protocol the same idea could
>     apply to make it more responsive to AQM.
>
>
> Interesting idea indeed. Drops in delay will of course also happen when,
> say, one flow stops, so it's not only an indicator of tail-drop (even
> though the probability of tail-drop may be higher if the drop in delay
> is "fishy").

A drop in delay alone isn't a trigger; it's a loss combined with a drop 
in delay.  I.e. from a packet train point of view for 30ms packets:

<- X - 31ms - X+1 - 31ms - X+2 - 31ms - X+4(!) - 31ms - X+5

That would be a fairly typical "signal" from a tail-drop router of 
buffer overflow - timing compressed by the queue. Also note the slope 
(increasing delay).  If it were:

<- X - 31ms - X+1 - 31ms - X+2 - 60ms - X+4(!) - 31ms - X+5

Then it would be much less "fishy".  Note this is most effective in a 
constrained-channel case, perhaps somewhat less so in a 
contention-for-channel case - but constrained channel is the "normal" 
unloaded access-link or WiFi bottleneck.  And in contention, it's still 
useful - you need to tune the amount a packet arrives "early", and I 
also added a rule to require multiple 'fishy' losses within a period 
(~2s) before it kicked in extra reduction, but that's a tuning/testing 
issue.

For LEDBAT, as a scavenger protocol, it may make sense for it to respond 
to all drops by reducing more than TCP would.  (Drops say either you 
have very short max queues (<~100ms), AQM, or other traffic has forced 
the queuing delay up to the limit - in either case you want to back off 
and yield.  So long as all the scavengers drop roughly similar amounts, 
they should be fair among themselves in the super-short-queue (or AQM) 
case.  In all other cases, LEDBAT would just get out of the way of 
foreground better/faster.  It might reduce efficiency slightly in some 
cases, but I suspect not much, and the only case we really care about 
(for LEDBAT) is a non-fully-loaded channel.

-- 
Randell Jesup
[email-masked]