Subject: [rtcweb] compressed codec-free webrtc?

From: Dave Taht <[email-masked]>
Date: Tue, 29 Oct 2013 11:33:07 -0700

In having my eyes glaze over at the codec debate I found myself
wondering to what extent anyone was pursuing truly low latency video
and audio, along the lines of what the lola project has been doing for
collaborative concerts.

See:

http://www.conts.it/artistica/lola-project

They ship raw audio and raw video, they actually use cameras where
they can get at scanlines and ship that (saving 16ms)

So I'm ignorant of what webrtc can do is there a codec selection (yuv?
48 bit audio? for the rawest video and audio possible?) All the extra
encoding steps we take today induce extra latency, and available
bandwidth continues to increase... and from a cc perspective if you
drop some bits from a scanline that's not a problem, but from audio
it's a huge deal....

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: Justin Uberti <[email-masked]>
Date: Tue, 29 Oct 2013 11:45:45 -0700

Uncompressed 720p30 (I420) is ~330 Mbps, so even though bandwidth continues
to increase, we're still not at the point where we no longer need codecs
(in the general case).

On Tue, Oct 29, 2013 at 11:33 AM, Dave Taht <[email-masked]> wrote:

> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
>

---

From: Christer Holmberg <[email-masked]>
Date: Tue, 29 Oct 2013 18:47:29 +0000

Hi Dave,

As far as the standards is concerned, you are allowed to implement and use whatever codec you want.

The discussion is about whether there should be a codec that everyone MUST implement.

Regards,

Christer

-----Alkuperäinen viesti-----
Lähettäjä: [email-masked] [mailto:[email-masked]] Puolesta Dave Taht
Lähetetty: 29. lokakuuta 2013 20:33
Vastaanottaja: [email-masked]; [email-masked]
Aihe: [rtcweb] compressed codec-free webrtc?

In having my eyes glaze over at the codec debate I found myself wondering to what extent anyone was pursuing truly low latency video and audio, along the lines of what the lola project has been doing for collaborative concerts.

See:

They ship raw audio and raw video, they actually use cameras where they can get at scanlines and ship that (saving 16ms)

So I'm ignorant of what webrtc can do is there a codec selection (yuv?
48 bit audio? for the rawest video and audio possible?) All the extra encoding steps we take today induce extra latency, and available bandwidth continues to increase... and from a cc perspective if you drop some bits from a scanline that's not a problem, but from audio it's a huge deal....

--
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Adam Roach <[email-masked]>
Date: Tue, 29 Oct 2013 14:32:03 -0500

On 10/29/13 13:33, Dave Taht wrote:
> 48 bit audio? for the rawest video and audio possible?)

In theory, what you're looking for is this: 
http://tools.ietf.org/html/rfc4175

However, the caveats in that document are pretty huge:

    It is important to note that uncompressed video can have immense
    bandwidth requirements (up to 270 Mbps for standard-definition video,
    and approximately 1 Gbps for high-definition video).  This is
    sufficient to cause potential for denial-of-service if transmitted
    onto most currently available Internet paths.

Given the risk of Destroying The Internet Forever[tm] if any major 
browser vendor implemented this, I don't expect you'll see much 
deployment any time soon.

/a

---

From: Dave Taht <[email-masked]>
Date: Tue, 29 Oct 2013 13:00:03 -0700

> http://tools.ietf.org/html/rfc4175

Thank you!

> Given the risk of Destroying The Internet Forever[tm] if any major browser
> vendor implemented this, I don't expect you'll see much deployment any time
> soon.

I don't see people doing videoconferences in the same qty as we do,
say, torrents (famous last words!). Certainly the panopticon might
take over the internet (but in that case delays from encoding should
be acceptable to our new robotic panoptic overloads, it's just the
pesky humans that have trouble with latency)

So, maybe running at rates like this would trouble the "Internet" a
bit now, but on internal networks, not so much.

270Mbits is a trivial amount of bandwidth for a modern switched
network. Homes and businesses that are wired are probably close to
universally switched GigE, so internal conversations could possibly
run at max speed and minimal latency.

Internet-wise, Gfiber delivers GigE, upcoming docsis standards go well
above 200Mbit, 802.11ac goes past a gig, 802.11ad goes to 7gig.

Removing a serious delay-inducing-encoding step makes things like echo
cancellation unneeded, and live lola-like-collaboration possible.

-- 
Dave Täht

---

From: Martin Thomson <[email-masked]>
Date: Tue, 29 Oct 2013 13:01:32 -0700

> http://tools.ietf.org/html/rfc4175

I'm surprised that this didn't come up in the codec discussions.  Of
course, you'd want to use FEC so that you could reduce the need for
loss concealment.

---

From: Leon Geyser <[email-masked]>
Date: Tue, 29 Oct 2013 23:14:37 +0200

Meanwhile where I live how long would it take to show one frame over a
256Kbit/s connection? :)
The MTI codec needs to be a lossy codec.

---

From: Randell Jesup <[email-masked]>
Date: Tue, 29 Oct 2013 23:00:17 -0400

>> they can get at scanlines and ship that (saving 16ms)

Yeah, such things are not the norm unless you build hardware at a low 
level; OS-level access is only at the frame level generally. Also, there 
are costs to pushing that many bits around, even in a local network.  
(Check your network interface overhead when pushing that many bits!)  
However: You could use slicing or equivalent to get almost the same 
delay characteristics at HUGELY lower datarates.

For example, use one-MB high slices with current-tech codecs or 
something very similar.  Compression will kinda suck (well, not so much 
if you let it reference previous slices in the same frame and later 
slices in previous frames), but for standard def (480/60p let's say) the 
inherent delay would be on the order of 0.5ms, perhaps with another 
0.5ms for encode time (max, otherwise we aren't keeping up with 60fps.  
For 30p it would be 2ms max.  I don't know about you, but I would have 
trouble seeing 2ms delay.  And how are you guaranteeeing you're not 
seeing a 16/33ms sync delay at the other end?  Are you somehow 
genlocking the displays?  Not *impossible*, but pretty unlikely.

Memory bandwidth >>> network bandwidth, even fiber, though realistic 
interfaces.   And display tech matters.  I think there must be some FUD 
out there about how compression delays everything and makes it 
problematic.  The one place where such tight latencies matter is audio 
(distributed bands, etc), and there speed-of-light (and 
speed-of-interfaces) are what trip you up generally.  There are video 
equivalents (coordinated remote dance, etc) - but you still have to 
solve the capture and display problems, and compression per se isn't the 
major problem (no new tech is needed, just correct use of current 
tech/codecs I think).

Brings to mind "Racing the beam".... ;-)

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 30 Oct 2013 05:15:59 +0000

Give app developers yuv, udp and native-speed js, and they won¹t need all
the webrtc plumbing at all. No chance of negotiation failure as icing. :)

Mo

On 10/29/13, 2:33 PM, Dave Taht <[email-masked]> wrote:

See:

-- 
Dave Täht

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 30 Oct 2013 06:19:21 +0100

Here at the MPEG meeting, one of my colleagues waxed lyrical about the
prospects of redesigning the CPU/memory/screen interfaces in order to
support the bandwidth requirements of getting 8K video from the decoder
to the screen.

Seems we may need a codec that decompresses at the group-of-pixel level,
or some new way of designing this, because normal memory buses can't
keep up with the data rate.

(8K is 7680 pixels wide by 4320 pixels tall (33.2 megapixels) according
to Wikipedia; at uncompressed 4:4:4, 3 bytes per pixel and 60 Hz, that's
23 Gbits/second, unless my calculator slipped a digit).

I think compression is going to be with us for a while.

On 10/30/2013 04:00 AM, Randell Jesup wrote:
> level; OS-level access is only at the frame level generally. Also,
> there are costs to pushing that many bits around, even in a local
> network.  (Check your network interface overhead when pushing that
> many bits!)  However: You could use slicing or equivalent to get
> almost the same delay characteristics at HUGELY lower datarates.
>
> something very similar.  Compression will kinda suck (well, not so
> much if you let it reference previous slices in the same frame and
> later slices in previous frames), but for standard def (480/60p let's
> say) the inherent delay would be on the order of 0.5ms, perhaps with
> another 0.5ms for encode time (max, otherwise we aren't keeping up
> with 60fps.  For 30p it would be 2ms max.  I don't know about you, but
> I would have trouble seeing 2ms delay.  And how are you guaranteeeing
> you're not seeing a 16/33ms sync delay at the other end?  Are you
> somehow genlocking the displays?  Not *impossible*, but pretty unlikely.
>
> interfaces.   And display tech matters.  I think there must be some
> FUD out there about how compression delays everything and makes it
> solve the capture and display problems, and compression per se isn't
> the major problem (no new tech is needed, just correct use of current
> tech/codecs I think).
>
>>> browser

-- 
Surveillance is pervasive. Go Dark.

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 30 Oct 2013 08:20:41 +0100

Hi,

I think using RAW video is completely out of the question for general
WebRTC usage, however I would note that the specification needed to
encapsulate RAW video is available as RTP payload formats, so you are
free to implement and offer them in signalling.

RTP Payload Format for Uncompressed Video
Which supports up to 32768 scan lines and pixels per line. It also
supports various samplings and color spaces.

http://tools.ietf.org/html/rfc4175
Data rate will depend on resolution, frame-rate and sampling.

RTP Payload Format for Society of Motion Picture and Television
Engineers (SMPTE) 292M Video

http://tools.ietf.org/html/rfc3497

Total data rate be either 1.485 Gbps or 1.485/1.001 Gbps.

Cheers

Magnus

-- 

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Matt Fredrickson <[email-masked]>
Date: Wed, 30 Oct 2013 10:08:48 -0500

> cancellation unneeded, and live lola-like-collaboration possible.

Echo cancellation will probably still be needed.  Audio as exposed to
userspace in the operating system is usually not low enough latency to
remove the need to do echo cancellation.  Packetization also induces delays
in the audio/video stack the necessitate echo cancellation.

Matthew Fredrickson