Subject: [rtcweb] FEC for audio?

From: Justin Uberti <[email-masked]>
Date: Mon, 19 May 2014 09:44:25 -0700

Should we specify generic FEC for audio as well as video? Opus has built-in
FEC, but it only works for the SILK portion of the payload, meaning that in
high bitrate situations, it won't do much for you. I don't know if it has
benefits over generic FEC.

Some application providers have mentioned that FEC helps with audio
intelligibility on cellular networks, so I think this is important to
support.

---

From: Peter Thatcher <[email-masked]>
Date: Mon, 19 May 2014 12:14:32 -0700

If we're already designing and implementing FEC for video, it seems like a
small step to also support it for audio.  I find the use case for
high-bitrate audio use compelling.  As an app, if I want to spend lots of
bits for really high-quality audio, I'd like the option to make sure FEC is
still enabled; I'm perhaps willing to spend the bits on it.

Is there a good reason not to support FEC for audio, if that's what the
application wants?

On Mon, May 19, 2014 at 9:44 AM, Justin Uberti <[email-masked]> wrote:

> Should we specify generic FEC for audio as well as video? Opus has
> built-in FEC, but it only works for the SILK portion of the payload,
> meaning that in high bitrate situations, it won't do much for you. I don't
> know if it has benefits over generic FEC.
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 19 May 2014 12:54:56 -0700

> Is there a good reason not to support FEC for audio, if that's what the

I can't think of one. I think we should support FEC for audio.

> On Mon, May 19, 2014 at 9:44 AM, Justin Uberti <[email-masked]
> <mailto:[email-masked]>> wrote:
>     meaning that in high bitrate situations, it won't do much for you. I
>     don't know if it has benefits over generic FEC.

The built-in FEC for the SILK layer was meant to cover the case where 
you wanted FEC with no more than one extra frame of latency, but did not 
want to double your bitrate. The reason we did not add built-in FEC for 
the CELT layer was that at the bitrates where CELT operates, we assumed 
traditional codec-agnostic FEC would be used, and thought it an adequate 
solution compared to the extra complexity a codec-specific solution 
would have required.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 19 May 2014 14:36:04 -0700

On Mon, May 19, 2014 at 12:54 PM, Timothy B. Terriberry <
[email-masked]> wrote:

> The built-in FEC for the SILK layer was meant to cover the case where you
> wanted FEC with no more than one extra frame of latency, but did not want
> to double your bitrate. The reason we did not add built-in FEC for the CELT
> layer was that at the bitrates where CELT operates, we assumed traditional
> codec-agnostic FEC would be used, and thought it an adequate solution
> compared to the extra complexity a codec-specific solution would have
> required.
>
>
Thanks, that's very helpful. This means that if the app says "do FEC", and
the remote side supports opus inband FEC, and the data rate is < XX kbps,
we'll use the inband FEC instead of generic FEC.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 19 May 2014 14:36:33 -0700

> we'll use the inband FEC instead of generic FEC.

Advice on the exact value of XX welcomed.

---

From: Silvia Pfeiffer <[email-masked]>
Date: Tue, 20 May 2014 09:05:14 +1000

Yes please (from a JS dev)! We often have enough bandwidth even on wireless
but packet loss and get broken audio. It's almost more important to have
this on audio than on video, but I'm keen to get both.

Cheers,
Silvia.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 19 May 2014 18:16:10 -0700

> Advice on the exact value of XX welcomed.

That is complicated, of course. I think if you're using SILK, you should 
probably always prefer the in-band FEC (if nothing else, it saves you a 
minimum of 54 bytes of header overhead over RFC5109), but that's a 
decision we make on a packet-by-packet basis.

However, libopus will use CELT at rates as low as 12 kbps (and always 
uses CELT for CBR rates below 8 kbps), and will use SILK/Hybrid at rates 
as high as 76 kbps, depending on the application libopus was configured 
with, our estimate of whether or not the content contains voice or 
music, the stereo width (if the signal is stereo), and hysteresis from 
previous mode decisions.

For a simple application, I'd probably enable 5109 for a mono voice 
stream encoded in VOIP mode at around 48-56 kbps (but I have not done 
any testing around these numbers). A more complex application could 
negotiate 5109 always, and then only transmit FEC for the packets that 
do not contain a SILK layer.

If you are using 5109, be careful to pass in the post-FEC recovery 
packet loss percentage to libopus when encoding, instead of the raw 
packet loss percentage. Above a certain loss rate (8% if I'm reading the 
current code correctly), libopus will force SILK usage in order to be 
able to use the built-in FEC.

When protecting CELT, you may also want to experiment with using ULP to 
protect only the beginning of the CELT payloads. We intentionally 
organized the packet so the most important information was at the front. 
The first 64-bits or so are enough to get the pitch filter and basic 
energy levels correct (though you'll probably want to send a little more 
to justify all the bitrate you spent on IP+UDP+RTP+FEC headers). As long 
as you get the energy levels correct, you cannot mess things up _too_ 
badly, even if the rest of the packet is random garbage. Make sure you 
pass the decoder a packet with the same size as the original, even if 
you only recover some of the data, as otherwise even the recovered data 
will not decode correctly.

---

From: Justin Uberti <[email-masked]>
Date: Tue, 20 May 2014 06:25:33 -0700

I don't think we would send a standalone FEC packet, given the overheads -
I would expect we would use RFC 2198 encoding and send 5109 FEC as
piggyback on the 'next' audio packet, similar to how I understand the
internal FEC works. This would have (5 + 10 + 4) * 8 = 152 bits of overhead
per packet, or ~8 Kbps when using 20 ms frames.

On Mon, May 19, 2014 at 6:16 PM, Timothy B. Terriberry <
[email-masked]> wrote:

> minimum of 54 bytes of header overhead over RFC5109), but that's a decision
> we make on a packet-by-packet basis.
>
> However, libopus will use CELT at rates as low as 12 kbps (and always uses
> CELT for CBR rates below 8 kbps), and will use SILK/Hybrid at rates as high
> as 76 kbps, depending on the application libopus was configured with, our
> estimate of whether or not the content contains voice or music, the stereo
> width (if the signal is stereo), and hysteresis from previous mode
> decisions.
>
> For a simple application, I'd probably enable 5109 for a mono voice stream
> encoded in VOIP mode at around 48-56 kbps (but I have not done any testing
> around these numbers). A more complex application could negotiate 5109
> always, and then only transmit FEC for the packets that do not contain a
> SILK layer.
>
> If you are using 5109, be careful to pass in the post-FEC recovery packet
> loss percentage to libopus when encoding, instead of the raw packet loss
> percentage. Above a certain loss rate (8% if I'm reading the current code
> correctly), libopus will force SILK usage in order to be able to use the
> built-in FEC.
>
> protect only the beginning of the CELT payloads. We intentionally organized
> the packet so the most important information was at the front. The first
> 64-bits or so are enough to get the pitch filter and basic energy levels
> correct (though you'll probably want to send a little more to justify all
> the bitrate you spent on IP+UDP+RTP+FEC headers). As long as you get the
> energy levels correct, you cannot mess things up _too_ badly, even if the
> rest of the packet is random garbage. Make sure you pass the decoder a
> packet with the same size as the original, even if you only recover some of
> the data, as otherwise even the recovered data will not decode correctly.
>
>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 20 May 2014 06:49:18 -0700

Justin Uberti wrote:
> I don't think we would send a standalone FEC packet, given the overheads
> - I would expect we would use RFC 2198 encoding and send 5109 FEC as
> internal FEC works. This would have (5 + 10 + 4) * 8 = 152 bits of
> overhead per packet, or ~8 Kbps when using 20 ms frames.

Yes, that would be much better.