Subject: [rtcweb] Options for DTLS warmup and directionality

From: Randell Jesup <[email-masked]>
Date: Thu, 23 Jul 2015 17:53:17 -0400

To expand on my idea at the mic today:

The current proposal assumes prior agreement to persistent permissions 
to work well (and may 'light up' the camera/mic while the call is still 
waiting for the user to accept, which would be concerning to users).

We should not require persistent permissions in order to "warm up" DTLS 
connections before "Accept" from the user.  We also must NOT spin up the 
microphone or camera before the user has agreed to accept the call.  We 
also want to avoid a=inactive and a=recvonly answers.

Proposed solution: leverage 'dummy' tracks and replaceTrack (at the JS 
API level) to avoid renegotiation.

We need to attach some sort of track to each one we expect to send on in 
order to make use of the warmed-up connection when Accept happens.  
Without persistent permissions, we need alternative streams.  We have a 
few options:

1) WebAudio + canvas.captureStream()
     * Generate a silent WebAudio stream and generate a 
MediaStream/MediaStreamTrack (s)
     * Generate a blank (or not!) MediaStream/MediaStreamTrack from 
canvas.captureStream() (s)
        Note you'd tell it to use low-or-no-framerate (see below for 
another option).
     * Use MediaStream constructors to combine the audio and video track(s)
     * Attach to PeerConnection, and createOffer() (will typically be 
sendrecv (or sendonly if the other side is recvonly)).
     * On Accept by the user, use RTPSender.replaceTrack() on all 
accepted tracks, and sending will begin immediately; no renegotiation 
needed if all are accepted.
     * No new APIs needed, no W3 work.  Requires implementing agreed-on 
APIs.

2) Add a new way to getUserMedia/whatever to create fake streams, ala 
fake:true in Firefox
     * Note that we'd want to modify Firefox's impl and not use 
fake:true (probably), since we generate a specific test pattern for video.
     * This is a W3 Media Capture issue, not IETF, but we should realize 
it requires aligning with them

3) Allow some form of fake track without creating one (perhaps related 
to Peter's comment at the mic), and then replaceTrack to null tracks.
     * Does not allow spinning up congestion.
     * Won't spin up other stuff (decoders, jitter buffers, etc). Some 
spec work required; may be dragons here.

I prefer option 1.  It requires no new APIs or protocol decisions, and 
no action by W3 WGs that are trying to finalize 1.0.   We can do this as 
soon as people implement what we've already agreed to.

As a side benefit, if you actually push video via the canvas stream via 
larger resolutions & framerates, it would spin up congestion control 
ahead of time as well - under control of the application. This could be 
a significant win.

-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email [email-masked]!  Way oo much spam

---

From: Peter Thatcher <[email-masked]>
Date: Thu, 23 Jul 2015 15:31:53 -0700

Here is my idea:

PeerConnection.addTrack(null) means "don't send anything until
PeerConnection.replaceTrack is called with a non-null track".

Then the code for the callee would something like the following (warning:
it's late at night; I may have goofed up something):

var pc = new PeerConnection(...);
var sender = pc.addTrack(null);

pc.setLocalDescriptioon(offer);
var answer = pc.createAnswer(offer);
pc.setRemoteDescription(answer);
// send answer

// getUserMedia and ring in parallel
// get a track when the user accepts
sender.setTrack(track);

Then you don't need any fake things.

It could spin up congestion and other stuff if we want it to.  There is an
RtpSender object there, after all.  But I'm not sure we actually want to.

On Thu, Jul 23, 2015 at 2:53 PM, Randell Jesup <[email-masked]>
wrote:

> The current proposal assumes prior agreement to persistent permissions to
> work well (and may 'light up' the camera/mic while the call is still
> Proposed solution: leverage 'dummy' tracks and replaceTrack (at the JS API
> level) to avoid renegotiation.
>
> order to make use of the warmed-up connection when Accept happens.  Without
> persistent permissions, we need alternative streams.  We have a few options:
>
>     * On Accept by the user, use RTPSender.replaceTrack() on all accepted
> tracks, and sending will begin immediately; no renegotiation needed if all
> are accepted.
>     * Note that we'd want to modify Firefox's impl and not use fake:true
> (probably), since we generate a specific test pattern for video.
>     * This is a W3 Media Capture issue, not IETF, but we should realize it
> requires aligning with them
>
> 3) Allow some form of fake track without creating one (perhaps related to
> Peter's comment at the mic), and then replaceTrack to null tracks.
>     * Won't spin up other stuff (decoders, jitter buffers, etc). Some spec
> work required; may be dragons here.
>
> I prefer option 1.  It requires no new APIs or protocol decisions, and no
> action by W3 WGs that are trying to finalize 1.0.   We can do this as soon
> as people implement what we've already agreed to.
>
> larger resolutions & framerates, it would spin up congestion control ahead
> of time as well - under control of the application. This could be a
> significant win.
>
> --
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Justin Uberti <[email-masked]>
Date: Fri, 24 Jul 2015 00:44:09 +0200

Peter's proposal pretty much matches what I just posted, but I think a
RTCRtpReceiver property is also needed to prevent caller->callee media
during ringing, which at a minimum would violate user expectations.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 24 Jul 2015 01:17:08 -0400

I'd love to be able to warm up congestion control, and not just DTLS.  
That requires sending data and allowing it to increase to probe the 
connection and open the congestion window, and preferably sending in 
both directions (i.e. a=sendrecv in the warm-up answer).

We'd need to find a way to generically pad RTP packets to a target size 
(since encoding a static/black image won't use enough bytes).  The 
obvious codec-agnostic mechanism would be a header extension.  Since 
they can be any 4-byte-multiple size, that would pretty much work up to 
~384K (1500 byte packets, 1/frame @30fps). However, we could run the 
fake video framerate arbitrarily high to hit any bandwidth target we need.

Alternatively (and *much* better) we could use a "fake" codec 
(a=rtpmap:111 fake/90000) which doesn't actually include any real data, 
just padding.  Advantages: no CPU code to encode/decode.  No dealing 
with header extensions.  Clear indication if the other side wants to do 
spinup (they can reject the codec if they don't want to).  No 
encoding/decoding CPU use.

The offer would include it as a least-preferred codec to indicate warmup 
support.  If answerer answers with fake as the accepted codec (and real 
codecs following), it's a warmup answer.  Both sides will send and 
receive fake data to spin up the congestion windows (and also the jitter 
buffers, etc).  On actual answer, the fake codec would be dropped (or 
dropped to bottom priority).   The answerer would start sending in a 
real codec, which would work since the warmup answer would have included 
that codec.  The offerer would see the incoming real-codec packets, and 
decode them.   When the real answer arrives, it will show the accepted 
codec has changed, and the offerer will install that answer and thus and 
start sending (real) media.

We still need a way to tell the system at the answerer side "I want to 
accept but it's a warm-up accept" (and perhaps if we want to spin up 
congestion control or not while waiting), and a way to say on the 
offerer side "I want to offer/use warmups".  For those API points see 
the previous options from myself and Peter - or simply a way to say 
createAnswer of a warmup answer, and have that agree to all tracks with 
fake codecs being primary.

I'll note that all of these methods would benefit from a mechanism to 
indicate the desired bitrate targets for each fake track; for solutions 
that use some sort of fake video input (canvas.captureStream/etc) it 
inherently allows the application to define the size and framerate, 
which are the primary knobs used by the browser for setting 'max total 
bitrate' for the track.   You don't want the application to set the 
bandwidth directly, you want it to define the parameters of the video 
stream so the system can set "reasonable" limits for that resolution & 
situation.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 24 Jul 2015 01:35:57 -0400

> during ringing, which at a minimum would violate user expectations.

If we want to answer a=sendrev offers with a=sendonly (regardless of 
this use-case), we need something like that.  However, we may not care 
much outside of this usecase.

I'd love to be able to just answer a=sendrecv in the warm-up answer, and 
in any case the answerer must indicate somehow in conjunction with the 
warmup answer that it's not a real user answer (I do not consider 
parsing all the directions for a=sendonly a reasonable way to know).  
Since the answer comes along with "this is warmup" (somehow), the 
offerer can see that and replaceTrack with fake (or null) tracks before 
SetRemoteDescription, and so not send media (or send only fake 
congestion-spinup data).

> On Fri, Jul 24, 2015 at 12:31 AM, Peter Thatcher <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Here is my idea:
>
>

Note that this would be problematic on the offerer side (is it audio or 
video?)  We'd have to figure out when this is valid and what it does 
when it id valid (in W3 land).  Also, perhaps we could use 
addTrack(null) as an indicator to accept a 'fake' codec as the primary 
codec (see previous email on congestion spinup).  This would enable (but 
not require) congestion spinup.

>
>
>     Then the code for the callee would something like the following
>     (warning: it's late at night; I may have goofed up something):
>

Another advantage of this (Peter's addTrack(null)) is that it doesn't 
send muted data (from webaudio or canvas.captureStream). This means when 
the user *does* answer, you swap to real sources and start encoding, so 
the offerer can key off incoming data to know it should start showing 
the data (before the final answer arrives). Note that the rtcpmap fake 
codec/payload idea would also work (the "decoder" would eat all data and 
never generate a frame of video, and would generate no audio or silent 
depending on how it works in the stack - then when the real answer 
happens, we switch to using a real encoder to encode real data, and it 
will end up decoding and showing).

>     It could spin up congestion and other stuff if we want it to. 
>     There is an RtpSender object there, after all. But I'm not sure we
>     actually want to.
>

Spinning congestion with 'null' tracks would be more complex - more API 
points probably (and needs a=sendrecv really, but per above we could).  
See previous message about how you'd do it.

-- 
Randell Jesup -- rjesup a t mozilla d o t com
Please please please don't email [email-masked]!  Way too much spam

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 24 Jul 2015 07:47:08 +0200

My initial reaction to this was I can see how it would work but I sort of hope we could get something that was easier for web developers and that did not use bandwidth to send the media before there was real media.  Good food for thought. 

> On Jul 23, 2015, at 11:53 PM, Randell Jesup <[email-masked]> wrote:
> 
> The current proposal assumes prior agreement to persistent permissions to work well (and may 'light up' the camera/mic while the call is still waiting for the user to accept, which would be concerning to users).
> 
> We should not require persistent permissions in order to "warm up" DTLS connections before "Accept" from the user.  We also must NOT spin up the microphone or camera before the user has agreed to accept the call.  We also want to avoid a=inactive and a=recvonly answers.
> 
> Proposed solution: leverage 'dummy' tracks and replaceTrack (at the JS API level) to avoid renegotiation.
> 
> We need to attach some sort of track to each one we expect to send on in order to make use of the warmed-up connection when Accept happens.  Without persistent permissions, we need alternative streams.  We have a few options:
> 
>    * Generate a silent WebAudio stream and generate a MediaStream/MediaStreamTrack (s)
>    * Generate a blank (or not!) MediaStream/MediaStreamTrack from canvas.captureStream() (s)
>       Note you'd tell it to use low-or-no-framerate (see below for another option).
>    * Attach to PeerConnection, and createOffer() (will typically be sendrecv (or sendonly if the other side is recvonly)).
>    * On Accept by the user, use RTPSender.replaceTrack() on all accepted tracks, and sending will begin immediately; no renegotiation needed if all are accepted.
>    * No new APIs needed, no W3 work.  Requires implementing agreed-on APIs.
> 
> 2) Add a new way to getUserMedia/whatever to create fake streams, ala fake:true in Firefox
>    * Note that we'd want to modify Firefox's impl and not use fake:true (probably), since we generate a specific test pattern for video.
>    * This is a W3 Media Capture issue, not IETF, but we should realize it requires aligning with them
> 
> 3) Allow some form of fake track without creating one (perhaps related to Peter's comment at the mic), and then replaceTrack to null tracks.
>    * Won't spin up other stuff (decoders, jitter buffers, etc). Some spec work required; may be dragons here.
> 
> I prefer option 1.  It requires no new APIs or protocol decisions, and no action by W3 WGs that are trying to finalize 1.0.   We can do this as soon as people implement what we've already agreed to.
> 
> As a side benefit, if you actually push video via the canvas stream via larger resolutions & framerates, it would spin up congestion control ahead of time as well - under control of the application. This could be a significant win.
> 
> --

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 24 Jul 2015 07:47:11 +0200

> On Jul 24, 2015, at 12:31 AM, Peter Thatcher <[email-masked]> wrote:
> 
> PeerConnection.addTrack(null) means "don't send anything until PeerConnection.replaceTrack is called with a non-null track".
> 

Would we need extra info to know if this was audio, video etc. I'm just thinking about the issues where the PC creates an answer without knowing anything about what is in the track then when the PC see the real track (after answer is sent), I'm wondering if we could end up with a case where what was in the track was not consistent with the answer that got sent. I don't have a concrete example of a problem here but I have a nagging feeling there might be problems here - particularly on things where the encoding is done in hardware.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 24 Jul 2015 09:50:06 +0200

On 24 July 2015 at 00:31, Peter Thatcher <[email-masked]> wrote:
> PeerConnection.addTrack(null)

I'm not entirely comfortable with the notion that we allow null
arguments.  Doing that could be a footgun.  Especially since it is
only to support what I consider to be a corner case (even though it
might be an important one).

---

From: Peter Thatcher <[email-masked]>
Date: Fri, 24 Jul 2015 01:54:14 -0700

Could the new flexfec payload type be used for padding?

On Thu, Jul 23, 2015 at 10:17 PM, Randell Jesup <[email-masked]>
wrote:

> connection and open the congestion window, and preferably sending in both
> directions (i.e. a=sendrecv in the warm-up answer).
>
> (since encoding a static/black image won't use enough bytes).  The obvious
> codec-agnostic mechanism would be a header extension.  Since they can be
> any 4-byte-multiple size, that would pretty much work up to ~384K (1500
> byte packets, 1/frame @30fps). However, we could run the fake video
> framerate arbitrarily high to hit any bandwidth target we need.
>
> just padding.  Advantages: no CPU code to encode/decode.  No dealing with
> header extensions.  Clear indication if the other side wants to do spinup
> (they can reject the codec if they don't want to).  No encoding/decoding
> CPU use.
>
> codecs following), it's a warmup answer.  Both sides will send and receive
> fake data to spin up the congestion windows (and also the jitter buffers,
> etc).  On actual answer, the fake codec would be dropped (or dropped to
> bottom priority).   The answerer would start sending in a real codec, which
> would work since the warmup answer would have included that codec.  The
> offerer would see the incoming real-codec packets, and decode them.   When
> the real answer arrives, it will show the accepted codec has changed, and
> the offerer will install that answer and thus and start sending (real)
> media.
>
> congestion control or not while waiting), and a way to say on the offerer
> side "I want to offer/use warmups".  For those API points see the previous
> options from myself and Peter - or simply a way to say createAnswer of a
> warmup answer, and have that agree to all tracks with fake codecs being
> primary.
>
> inherently allows the application to define the size and framerate, which
> are the primary knobs used by the browser for setting 'max total bitrate'
> for the track.   You don't want the application to set the bandwidth
> directly, you want it to define the parameters of the video stream so the
> system can set "reasonable" limits for that resolution & situation.
>

---

From: Peter Thatcher <[email-masked]>
Date: Fri, 24 Jul 2015 01:57:03 -0700

Would you be comfortable with either addTrack("audio") or
createRtpSender("audio")?

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 24 Jul 2015 11:00:54 +0200

On Fri, Jul 24, 2015 at 10:57 AM, Peter Thatcher <[email-masked]>
wrote:

> Would you be comfortable with either addTrack("audio")
>

This seems awful.

> or createRtpSender("audio")?
>

This seems OKish

-Ekr

> On Fri, Jul 24, 2015 at 12:50 AM, Martin Thomson <[email-masked]
> > wrote:
>

---

From: Luis López Fernández <[email-masked]>
Date: Fri, 24 Jul 2015 11:47:19 +0200

This is an interesting possibility. Using some kind of fec as padding could make possible to avoid using a fake codec allowing receivers to behave normally and without requiring implementing a different behavior for the fake codec data. In addition, this protection of the encoded static/black image may avoid PLIs in case of severe packet loss when hitting a BW during the probe. This could be even used as a probe mechanism for BW estimation with the call fully open, but this is another story.
El 24/07/2015, a las 10:54, Peter Thatcher <[email-masked]> escribió:

> On Thu, Jul 23, 2015 at 10:17 PM, Randell Jesup <[email-masked]> wrote:
> I'd love to be able to warm up congestion control, and not just DTLS.  That requires sending data and allowing it to increase to probe the connection and open the congestion window, and preferably sending in both directions (i.e. a=sendrecv in the warm-up answer).
> 
> We'd need to find a way to generically pad RTP packets to a target size (since encoding a static/black image won't use enough bytes).  The obvious codec-agnostic mechanism would be a header extension.  Since they can be any 4-byte-multiple size, that would pretty much work up to ~384K (1500 byte packets, 1/frame @30fps). However, we could run the fake video framerate arbitrarily high to hit any bandwidth target we need.
> 
> Alternatively (and *much* better) we could use a "fake" codec (a=rtpmap:111 fake/90000) which doesn't actually include any real data, just padding.  Advantages: no CPU code to encode/decode.  No dealing with header extensions.  Clear indication if the other side wants to do spinup (they can reject the codec if they don't want to).  No encoding/decoding CPU use.
> 
> The offer would include it as a least-preferred codec to indicate warmup support.  If answerer answers with fake as the accepted codec (and real codecs following), it's a warmup answer.  Both sides will send and receive fake data to spin up the congestion windows (and also the jitter buffers, etc).  On actual answer, the fake codec would be dropped (or dropped to bottom priority).   The answerer would start sending in a real codec, which would work since the warmup answer would have included that codec.  The offerer would see the incoming real-codec packets, and decode them.   When the real answer arrives, it will show the accepted codec has changed, and the offerer will install that answer and thus and start sending (real) media.
> 
> We still need a way to tell the system at the answerer side "I want to accept but it's a warm-up accept" (and perhaps if we want to spin up congestion control or not while waiting), and a way to say on the offerer side "I want to offer/use warmups".  For those API points see the previous options from myself and Peter - or simply a way to say createAnswer of a warmup answer, and have that agree to all tracks with fake codecs being primary.
> 
> I'll note that all of these methods would benefit from a mechanism to indicate the desired bitrate targets for each fake track; for solutions that use some sort of fake video input (canvas.captureStream/etc) it inherently allows the application to define the size and framerate, which are the primary knobs used by the browser for setting 'max total bitrate' for the track.   You don't want the application to set the bandwidth directly, you want it to define the parameters of the video stream so the system can set "reasonable" limits for that resolution & situation.
>

---

From: Justin Uberti <[email-masked]>
Date: Fri, 24 Jul 2015 11:51:54 +0200

I tend to think BWE warmup is out of scope for this discussion. It's not
clear that the callee wants to receive perhaps substantial amounts of
traffic prior to having accepted the call.

On Fri, Jul 24, 2015 at 11:47 AM, Luis López Fernández <[email-masked]>
wrote:

> This is an interesting possibility. Using some kind of fec as padding
> could make possible to avoid using a fake codec allowing receivers to
> behave normally and without requiring implementing a different behavior for
> the fake codec data. In addition, this protection of the encoded
> static/black image may avoid PLIs in case of severe packet loss when
> hitting a BW during the probe. This could be even used as a probe mechanism
> for BW estimation with the call fully open, but this is another story.
>

---

From: Justin Uberti <[email-masked]>
Date: Fri, 24 Jul 2015 11:50:41 +0200

How about:

track = new MediaStreamTrack("audio")
addTrack(track)

or

track = new MediaStreamTrack();
track.kind = "audio";
addTrack(track)

On Fri, Jul 24, 2015 at 11:00 AM, Eric Rescorla <[email-masked]> wrote:

>> On Fri, Jul 24, 2015 at 12:50 AM, Martin Thomson <
>> [email-masked]> wrote:
>>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 24 Jul 2015 12:13:45 +0200

> track = new MediaStreamTrack();

I would like to keep track.kind readonly.

---

From: Peter Thatcher <[email-masked]>
Date: Fri, 24 Jul 2015 03:15:11 -0700

I prefer a blank RtpSender over a blank Track, and thus prefer
createRtpSender("audio"), but I could live with "new
MediaStreamTrack('audio')".  I wouldn't want the second one where you set
the kind because there's a time period in which you have a track with no
kind, which is strange, even for a fake one.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 24 Jul 2015 06:29:31 -0400

On 7/24/2015 6:15 AM, Peter Thatcher wrote:
> MediaStreamTrack('audio')".  I wouldn't want the second one where you 
> set the kind because there's a time period in which you have a track 
> with no kind, which is strange, even for a fake one.

If you have new MediaStreamTrack("audio"), you need to define what this 
means for *other* sinks/etc for MediaStreamTracks...  media elements, 
webaudio, combined in MediaStream constructors, etc... which is an 
argument for createRtpSender("audio").

Note that from JS's point of view, track = new MediaStreamTrack(); 
track.kind = "audio" is all in the same 'run' between stable states, so 
it's not really supposed to be "observable".  However, what if the JS 
doesn't do that... you can have a track that's neither audio nor video...

>
> On Fri, Jul 24, 2015 at 2:50 AM, Justin Uberti <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     How about:
>
>     On Fri, Jul 24, 2015 at 11:00 AM, Eric Rescorla <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>
>         On Fri, Jul 24, 2015 at 10:57 AM, Peter Thatcher
>         <[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: Stefan Holmer <[email-masked]>
Date: Fri, 24 Jul 2015 10:51:58 +0000

Luis, for what you are talking about there exists a draft in the rmcat
working group: https://tools.ietf.org/html/draft-singh-rmcat-adaptive-fec-01

I tend to agree that it may not be a good idea to blast packets to the
other end before accepting the call.

/Stefan

On Fri, Jul 24, 2015 at 11:52 AM Justin Uberti <[email-masked]> wrote:

> On Fri, Jul 24, 2015 at 11:47 AM, Luis López Fernández <[email-masked]
> > wrote:
>

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 24 Jul 2015 13:05:24 +0200

I have added this bug:

https://github.com/w3c/mediacapture-main/issues/221

to the W3C github in order to track creating "naked" MediaStreamTracks.
Please comment there, or on [email-masked].

---

From: Peter Thatcher <[email-masked]>
Date: Fri, 24 Jul 2015 04:19:56 -0700

I sent an email to the W3C list initiating discussion about what API points
we need.  I think we should continue all discussion there.

And I think the bug is more than just "Add naked tracks", and that title is
already assuming a particular solution to the problem.