Subject: [rtcweb] JSEP question: How to set up simulcast for server-originated calls?

From: Harald Alvestrand <[email-masked]>
Date: Sun, 4 Nov 2018 09:13:28 +0100

In JSEP, we have the following text:

   In addition, JSEP does not provide a mechanism to handle an incoming
   offer requesting simulcast from the JSEP endpoint.  This means that
   setting up simulcast in the case where the JSEP endpoint receives the
   initial offer requires out-of-band signaling or SDP inspection.

In WEBRTC, we have been grappling with the problem of simulcast setup,
and it seems that we have no obvious API surface that can be used for
this purpose - creating transceivers (as we do for outgoing) doesn't fit
the bill, because incoming SDP causes transceivers to be created wihout
interacting with the API surface.

I tried to drum up a test that would create such a setup based on SDP,
but found that in conformance with JSEP, this doesn't work.

Can someone tell me why we decided to not allow an incoming offer with
a=simulcast and a=rid ... recv in it to generate outgoing simulcast
capability?

The test in question:

https://github.com/web-platform-tests/wpt/pull/13902

Harald

-- 
Surveillance is pervasive. Go Dark.

---

From: Nils Ohlmeier <[email-masked]>
Date: Sun, 4 Nov 2018 15:34:35 +0700

> On 4Nov, 2018, at 15:13, Harald Alvestrand <[email-masked]> wrote:
> 

I can’t answer your question of why not.
But it is working in Firefox as this test shows 
https://searchfox.org/mozilla-central/source/dom/media/tests/mochitest/test_peerConnection_simulcastAnswer.html#59 <https://searchfox.org/mozilla-central/source/dom/media/tests/mochitest/test_peerConnection_simulcastAnswer.html#59>

I would have to look into the details, but I think for it to work you have to setup the RTP senders before setting the Offer (which is arguably not a nice API design). Which usually is not a big problem as the server which wants to receive simulcast knows how many RID’s it wants to get.

Best
  Nils Ohlmeier

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:03:20 +0100

> interacting with the API surface.

That's the exact problem: the fact that media parameters are given to
the other party by sharing a blob that the receiver must consume
before it even knows what it contains and, once consumed, it must rely
on "events" fired out of context (cannot correlate the received SDP
blob and peerconnection/stream events generated after consuming it).

And worse, once such a SDP blob is received and blindly consumed,
"things" happen without app knowledge or consent. For example and as
said by Nils, Firefox can conform to simulcast settings on receipt of
a remote SDP offer, but if we think about it, it means that the remote
decides in behalf of us (we cannot decide how to send simulcast but,
instead, the remote can force us to send simulcast as it wishes (and
not how we wish) by just sending us a blob reoffer.

The problem is the SDP as "API surface" as it's been always. IMHO we
shouldn't waste more energies adding more hacks to the current API to
fight the SDP O/A nature and, instead, we should move to a SDP-less
specification ASAP in which the remote party cannot blindly mandate
the local peer what to send at any time.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:36:58 +0100

I agree with Iñaki that the remote side should not be able to start
simulcast locally without app involvement.

IMHO best would be to keep things as they are, do not allow to receive a
simulcast offer. Instead, send a remote offer without simulcast from sfu,
add transceiver with simulcast encodings,  create answer without simulcast
m line, and trigger renegotiate needed event so the app creates a new sdp
offer sdp with the simulcast m line this time.

Convoluted but better than opening a new rabbit hole.

Best regards
Sergio

El dom., 4 nov. 2018 23:03, Iñaki Baz Castillo <[email-masked]> escribió:

> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:42:31 +0100

On Sun, 4 Nov 2018 at 23:37, Sergio Garcia Murillo
<[email-masked]> wrote:
>
> I agree with Iñaki that the remote side should not be able to start simulcast locally without app involvement.
>
> IMHO best would be to keep things as they are, do not allow to receive a simulcast offer. Instead, send a remote offer without simulcast from sfu, add transceiver with simulcast encodings,  create answer without simulcast m line, and trigger renegotiate needed event so the app creates a new sdp offer sdp with the simulcast m line this time.

Unrelated, but funny fact:

* Use unified-plan in Chrome.

* Enable simulcast in a m=video section via offer munging (append
additional a=ssrc lines, etc).

* Later remove the video track via pc.removeTrack(transceiver.sender).

* Multiple a=ssrc lines are still there.

* If later the same transceiver is re-enabled for sending a (new)
video track (transceiver.sender.replaceTrack(newTrack)) simulcast is
automatically enabled.

Not sure if expected, and not sure what should happen when using the
standard simulcast API sender.setParameters(), but IMHO it's funny.

Oh, and call transceiver.sender.replaceTrack(audioTrack) and
auto-magically the m=section will change from m=video to m=audio XDD

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 4 Nov 2018 17:47:19 -0500

Nils said:

" would have to look into the details, but I think for it to work you have
to setup the RTP senders before setting the Offer (which is arguably not a
nice API design). Which usually is not a big problem as the server which
wants to receive simulcast knows how many RID’s it wants to get."

[BA] Typically the Offerer is a conference server, not a browser. And it is
sending an Offer to receive simulcast, not to send it.  The browser does
have to setup its RtpSender to send simulcast based on the Offer from the
conference server, once applied. Some aspects of this seem clear - the
number of encodings should correspond to the number in the Offer, and the
RIDs should match.  But some other aspects are not as clear - in the test,
the Offer contained max-fr in the a=rid:X recv lines.  In the FF
implementation do these result in setting of maxFramerate attributes in the
RtpSender?

"
On Sun, Nov 4, 2018 at 3:34 AM Nils Ohlmeier <[email-masked]> wrote:

> https://searchfox.org/mozilla-central/source/dom/media/tests/mochitest/test_peerConnection_simulcastAnswer.html#59
>
> I would have to look into the details, but I think for it to work you have
> to setup the RTP senders before setting the Offer (which is arguably not a
> nice API design). Which usually is not a big problem as the server which
> wants to receive simulcast knows how many RID’s it wants to get.
>
> Best
>   Nils Ohlmeier

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 4 Nov 2018 17:50:30 -0500

Sergio said:

"I agree with Iñaki that the remote side should not be able to start
simulcast locally without app involvement.
IMHO best would be to keep things as they are, do not allow to receive a
simulcast offer. Instead, send a remote offer without simulcast from sfu,
add transceiver with simulcast encodings,  create answer without simulcast
m line, and trigger renegotiate needed event so the app creates a new sdp
offer sdp with the simulcast m line this time."

[BA] That won't work because the conferencing server is the only one that
knows how many participants are in the conference, so it needs to send the
Offer.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:52:38 +0100

On Sun, 4 Nov 2018 at 23:47, Bernard Aboba <[email-masked]> wrote:
> [BA] Typically the Offerer is a conference server, not a browser. And it is sending an Offer to receive simulcast, not to send it.  The browser does have to setup its RtpSender to send simulcast based on the Offer from the conference server, once applied. Some aspects of this seem clear - the number of encodings should correspond to the number in the Offer, and the RIDs should match.  But some other aspects are not as clear - in the test, the Offer contained max-fr in the a=rid:X recv lines.  In the FF implementation do these result in setting of maxFramerate attributes in the RtpSender?

Hi Bernard, before entering into implementation details, do you really
agree that the remote should be able to force the local peer what and
how to send media?

Also, you are assuming the scenario in which the conference server
offers a peer to receive simulcast from him. Where does such an
assumption come from? We don't need that assumption in WebRTC land. If
the conference *application* wants to ask peers to send simulcast can
be achieved by tons of ways (via custom messages or whatever), we
should not reduce everything to the "SIP world" in which everything is
expressed by sending a SDP.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:55:43 +0100

How does the app set up the rtp sender?

How does the app knows that the offer has simulcast enabled and the rids
without sdp inspection?

Best regards
Sergio

El dom., 4 nov. 2018 23:47, Bernard Aboba <[email-masked]>
escribió:

> [BA] Typically the Offerer is a conference server, not a browser. And it
> is sending an Offer to receive simulcast, not to send it.  The browser does
>> a=simulcast and a=rid .... recv in it to generate outgoing simulcast
>> capability?
>>
>> The test in question:
>>
>> I would have to look into the details, but I think for it to work you
>> have to setup the RTP senders before setting the Offer (which is arguably
>> not a nice API design). Which usually is not a big problem as the server
>> which wants to receive simulcast knows how many RID’s it wants to get.
>>
>> Best
>>   Nils Ohlmeier

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:56:03 +0100

On Sun, 4 Nov 2018 at 23:50, Bernard Aboba <[email-masked]> wrote:
>
> Sergio said:
>
> "I agree with Iñaki that the remote side should not be able to start simulcast locally without app involvement.
> IMHO best would be to keep things as they are, do not allow to receive a simulcast offer. Instead, send a remote offer without simulcast from sfu, add transceiver with simulcast encodings,  create answer without simulcast m line, and trigger renegotiate needed event so the app creates a new sdp offer sdp with the simulcast m line this time."
>
> [BA] That won't work because the conferencing server is the only one that knows how many participants are in the conference, so it needs to send the Offer.

Yes, and when other peers want to join the conference, the server can
provide them with a JSON like this:

{
  useSimulcast : true,
  simulcastStreams: [
    { rid: 1, priority: "high", maxBitrate: 100000 },
    { rid2 1, priority: "medium", maxBitrate: 300000 }
  ]
}

so the peer creates a peerconnection, adds a video transceiver, sets
the corresponding encodings into its RtpSender and we are done.

We don't need to reduce/constraint the media signaling preferences to SDP land.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Sun, 4 Nov 2018 23:58:23 +0100

On Sun, 4 Nov 2018 at 23:56, Sergio Garcia Murillo
<[email-masked]> wrote:
> How does the app knows that the offer has simulcast enabled and the rids without sdp inspection?

Ironically the app would know all those details *after* having
consuming the remote SDP and after having enabled simulcast as the
remote wishes. This is, without app consent and without app knowledge.

This is a big NO-WAY.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 4 Nov 2018 18:30:59 -0500

Inaki said:

"Hi Bernard, before entering into implementation details, do you really
agree that the remote should be able to force the local peer what and
how to send media?"

[BA] The remote can't force the local peer to send media. The local peer
still has to configure the direction and encodings to describe how and
whether to send simulcast. The main simulcast information communicated in
the Offer is the maximum number of streams the conferencing server can
accept and the RID names.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 00:33:37 +0100

On Mon, 5 Nov 2018 at 00:31, Bernard Aboba <[email-masked]> wrote:
> [BA] The remote can't force the local peer to send media. The local peer still has to configure the direction and encodings to describe how and whether to send simulcast. The main simulcast information communicated in the Offer is the maximum number of streams the conferencing server can accept and the RID names.

Once the remote (the conf server) sends its SDP offer, the local peer
can just blindly trust it and consume it via
pc.setRemoteDescription(), and once done, the party has already
started:

- The transceiver has been automatically created.
- If the local pc had a video track, a simulcast enabled RtpSender has
been automatically created.
- etc etc, everything without app knowledge/consent.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 4 Nov 2018 18:34:53 -0500

Sergio said:

"Ironically the app would know all those details *after* having
consuming the remote SDP and after having enabled simulcast as the
remote wishes. This is, without app consent and without app knowledge.

This is a big NO-WAY."

[BA] All the Offer communicates is that the conferencing server can receive
up to a maximum number of simulcast streams and the RID names. So
presumably the application still has to call
transceiver.sender.setParameters() to set the other details of the
simulcast transmission (such as whether streams are active, their
resolution scale, maximum framerate, etc.).

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 00:38:09 +0100

On Mon, 5 Nov 2018 at 00:35, Bernard Aboba <[email-masked]> wrote:
> [BA] All the Offer communicates is that the conferencing server can receive up to a maximum number of simulcast streams and the RID names. So presumably the application still has to call transceiver.sender.setParameters() to set the other details of the simulcast transmission (such as whether streams are active, their resolution scale, maximum framerate, etc.).

As told above, there is no way for the app to decide that upon receipt
of the remote offer, since there is no API hook to set those simulcast
and encodings parameters. In fact, Firefox obeys the simulcast
parameters of the remote offer automatically and start sending all the
simulcast streams as requested by the remote offerer.

So, the thing here is whether WebRTC 1.0 (SDP based) deserves even
more API hacks to enable this feature (manage sending simulcast upon
receipt of a remote offer) or if's just better to leave this feature
unsupported and wait for a proper non SDP based API (WebRTC-NG?).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 4 Nov 2018 19:06:36 -0500

On Nov 4, 2018, at 6:38 PM, Iñaki Baz Castillo <[email-masked]> wrote:
> 
> As told above, there is no way for the app to decide that upon receipt of the remote offer, since there is no API hook to set those simulcast and encodings parameters.

[BA] There is sender.setParameters(), which can turn simulcast encodings on/off or set other parameters relating to simulcast.

> In fact, Firefox obeys the simulcast
> parameters of the remote offer automatically and start sending all the simulcast streams as requested by the remote offerer.

[BA] That doesn’t seem like a good idea. For example, the application could start sending 3 simulcast streams, all at the same bitrate, resolution or framerate. Might make more sense for the default for Offer-created streans to be active: false until the application decides differently with setParameters(). 

> wait for a proper non SDP based API (WebRTC-NG?).

[BA] No need to wait for a non-SDP API, it’s been implemented and already handles this case quite nicely ;)

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 01:08:18 +0100

> [BA] There is sender.setParameters(), which can turn simulcast encodings on/off or set other parameters relating to simulcast.

You receive a SDP blob, pass it to pc.setRemoteDescription() and...
and right now you are already sending simulcast streams.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Nils Ohlmeier <[email-masked]>
Date: Mon, 5 Nov 2018 07:21:49 +0700

> On 5Nov, 2018, at 06:38, Iñaki Baz Castillo <[email-masked]> wrote:
> 

Here is a fiddle which demonstrates that just putting simulcast into the offer, but not calling setEncodings() does not activate simulcast: https://jsfiddle.net/x6qgab87/ <https://jsfiddle.net/x6qgab87/>
So the other side can’t force simulcast on the browser. If you provide me with a demo which shows that it is possible, then that is a bug I’ll happily fix for you.

Best
  Nils Ohlmeier

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 01:26:28 +0100

(On Mon, 5 Nov 2018 at 01:21, Nils Ohlmeier <[email-masked]> wrote:
>
> Here is a fiddle which demonstrates that just putting simulcast into the offer, but not calling setEncodings() does not activate simulcast: https://jsfiddle.net/x6qgab87/

Ok, that's new. I remember many months ago when I reported this (I was
in fact producing simulcast based on remote SDP offer) and it was not
necessary to call sender.setParameters({ encodings }) after sRD and
before createOffer.

Ok, this behavior makes more sense. BUT: it's not feasible because you
need to use hardcoded/fixed rid values (or manually do SDP parsing).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 01:37:12 +0100

there could be a way to overcome this by setting all the send encodings
active flag to false when receiving a simulcast offer. Then app can do a
getParameters, get layer num and rids, configure the settings and enable
the desired ones by setting active to true.

Best regards
Sergio

El lun., 5 nov. 2018 1:26, Iñaki Baz Castillo <[email-masked]> escribió:

> > Here is a fiddle which demonstrates that just putting simulcast into the
> offer, but not calling setEncodings() does not activate simulcast:
> https://jsfiddle.net/x6qgab87/
> > So the other side can’t force simulcast on the browser. If you provide
> me with a demo which shows that it is possible, then that is a bug I’ll
> happily fix for you.
>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 01:41:28 +0100

Mmm, true, RtpReceiver does also have a getParameter() so the app can
get the desired list of rids.
On Mon, 5 Nov 2018 at 01:37, Sergio Garcia Murillo
<[email-masked]> wrote:
>
> there could be a way to overcome this by setting all the send encodings active flag to false when receiving a simulcast offer. Then app can do a getParameters, get layer num and rids, configure the settings and enable the desired ones by setting active to true.
>
> Best regards
> Sergio
>

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 03:57:29 +0100

On 11/04/2018 11:36 PM, Sergio Garcia Murillo wrote:
> IMHO best would be to keep things as they are, do not allow to receive
> a simulcast offer. Instead, send a remote offer without simulcast from
> sfu, add transceiver with simulcast encodings,  create answer without
> simulcast m line, and trigger renegotiate needed event so the app
> creates a new sdp offer sdp with the simulcast m line this time.
The question at hand is:

If the remote SDP indicates that the remote end is willing to receive 3
simulcast layers, and the browser is able to send 3 simulcast layers,
how many simulcast encodings should be represented in the
sender.getParameters() returned after the negotiation completes - and
what are the implications of that?

Remember that we already decided (in WEBRTC) that the number of
encodings in the sender.getParameters() is the way the browser tells the
Javascript what the maximum number of simulcast encodings it can
configure is.

At no point did I want to address how many of those layers should be
enabled; I think the same thing as we do in other places (enable 1 layer
by default, others must be enabled by setParameters()) is fine.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 04:01:59 +0100

> creates a new sdp offer sdp with the simulcast m line this time.

The problem is that there is no way under current WEBRTC procedures to
increase the number of encodings on a sender after creating the sender.

we're already in the rabbit hole.

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 5 Nov 2018 12:16:31 +0700

> On Nov 5, 2018, at 5:52 AM, Iñaki Baz Castillo <[email-masked]> wrote:
> 

I agree with Bernard. 

It's not about force something to do something - the app can always do whatever it wants with offers and answers -  it's about the local side being able to interop with a changing remote side without having to worry about anything local. Every time the browsers add new features, it's pretty awesome if my application gets them without even worrying about them if there is no need to worry about them.

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 07:40:56 +0100

That would create a new rtpsender with all the encodings and not reuse the
initial transceiver as it dont support simulcast.

It should be supported under current specs.

Best regards
Sergio

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 09:03:16 +0100

On 11/05/2018 07:40 AM, Sergio Garcia Murillo wrote:
> That would create a new rtpsender with all the encodings and not reuse
> the initial transceiver as it dont support simulcast.

We don't have a means of replacing the transceiver associated with a
media section. So replacing it would mean rejecting the already
negotiated media section and adding a new media section that has exactly
the same properties, except that it has the properties of the media
section that the original offer requested.

I must admit that I have problems understanding what the problem is with
adding multiple (disabled) encodings on the created transceiver.
> El lun., 5 nov. 2018 4:02, Harald Alvestrand <[email-masked]
> <mailto:[email-masked]>> escribió:
>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 10:03:30 +0100

On 05/11/2018 9:03, Harald Alvestrand wrote:
>> That would create a new rtpsender with all the encodings and not 
>> reuse the initial transceiver as it dont support simulcast.
>
> negotiated media section and adding a new media section that has 
> exactly the same properties, except that it has the properties of the 
> media section that the original offer requested.
I am not talking about replacing a transceiver but adding a new one and 
renegotiate. The initial O/A will drop the simulcast attribute on the 
answer and create a recvonly transceiver, and then create a new 
transceiver with simulcast encoding that will require a O/A.
>
> I must admit that I have problems understanding what the problem is 
> with adding multiple (disabled) encodings on the created transceiver.

My fears are that this change may not be self contained within a single 
chapter/paragraph but spawn over all the spec and introducing new edge 
cases we have to solve in sections that are already closed.

Best regards
Sergio

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 12:45:40 +0100

On 11/05/2018 10:03 AM, Sergio Garcia Murillo wrote:
> I am not talking about replacing a transceiver but adding a new one
> and renegotiate. The initial O/A will drop the simulcast attribute on
> the answer and create a recvonly transceiver, and then create a new
OK, so we have the same picture in mind of what will happen in your
proposal. We end up with an useless transceiver and a new transceiver
that is suggested by the browser end, in a new media section; it's
impossible to use the media section that the (initiating) server proposed.

That seems ugly.
> My fears are that this change may not be self contained within a
> single chapter/paragraph but spawn over all the spec and introducing
> new edge cases we have to solve in sections that are already closed.

Are there closed sections?

I do fear that simulcast will continue to affect parts of the spec that
we don't expect it to. But we don't have the luxury of specifying it
halfway; either we rip it out altogether, or we specify it in enough
detail for interoperable implementation.

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 12:54:46 +0100

> impossible to use the media section that the (initiating) server proposed.

It will not be useless, just recvonly. Many of the SFU devs that I know 
(including myself) plan to use sendonly/recvonly transceivers anyway to 
avoid having to match transceivers for incoming and outgoing tracks.

Best regards

Sergio

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 13:06:04 +0100

> what are the implications of that?

This is a very good point. Yes, even if we inspect
receiver.getParameters() after pc.setRemoteDescription(offer), we'd
just get the number of encodings wished by the remote, and having to
correlate such a number with the local limit (do we have an API to
know the local limit before calling sender.setParameters()?) would be
an implementation and usage nightmare.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 13:11:55 +0100

> impossible to use the media section that the (initiating) server proposed.

So:

- receive a remote offer with a single m=video section requesting
local simulcast
- reply with a=recvonly (fxxx you, conference server)
- reoffer with a new m=video with "correlated" simulcast settings
- have the conference server like that

And... all this party instead of just telling the client (via custom
messaging before it "joins" the conference) the desired simulcast
settings? :)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 13:14:23 +0100

If you excluded the word "interop" from any rationale, life would be
much better, specially here where such an interop:

1) does not exist
2) is not needed

To be clear: No app in the world is or will be NEVER ready to enable
simulcast auto-magically just because it eventually receives a
different SDP offer from the remote. The app developers would already
figured out different ways to enable simulcast that would just not
allow such an auto-magic "upgrade".

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 13:25:30 +0100

> messaging before it "joins" the conference) the desired simulcast
Not sure if you are responding to  Harald or to me.  If it is to me, my 
"do nothing" proposal to enable simulcast when the offer is sent by the 
SFU (just in case anyone wants to do it):

-SFU sends an offer with all the available streams/tracks (no simulcast 
m-line)
-Browser reply with recvonly on all m-lines
-Browsers reoffer with an additional m-line (sendonly?) with simulcast
-SFU answers

No change to the spec at the cost of an additional round trip. Note that 
this is almost the same case than if the browser sends and initial offer 
with simulcast, sfu answers with recvonly, and then re-offers with all 
the available streams/track.

Also, note that many of SFU devs (at minimum Iñaki and me)  are not even 
sending SDP on the wire but doing renegotiations locally.

Best regards
Sergio

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 13:28:30 +0100

> sending SDP on the wire but doing renegotiations locally.

Ok, let me clarify:

Of course your approach would work, and also mine (signal server
desired recv simulcast by other means but SDP before the client
joins), but I strongly believe that people wishing to enable simulcast
in clients by just receiving an offer from the server is because they
have a fixed architecture in which the server sends the offer, expect
a "full featured" answer and that's all. Probably their servers do not
allow receiving reoffers from clients (you know the pain of managing
both send and recv streams within a single m= section when you want to
control sending/receiving codecs, etc).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 14:07:07 +0100

> detail for interoperable implementation.

I agree to that. If we are able to find an "easy" (i.e. not introducing 
too many changes) I would not have any issues adding support to that.

Something like:

- Browser adds a track via addTrack

- Browser call SRD with a recv simulcast m-line offer

- The transceiver is created with one send encoding per rid offered by 
the SFU with default values and all send encodings except first one with 
active=false (this would be the only required change)

----- browser encodes/sends a single rtp stream, same as if the remote 
offer was a non simulcast stream--

-Browser calls sender.getParameters(), modifies the desired send 
encoding parameters ( scaleResolutionDownBy,  maxFramerate, maxBitrate, 
etc) and enables them by setting active=true

Would this approach fulfill the requirements?

Best regards

Sergio

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 14:14:39 +0100

As Harlad said before, sender.getParameters() should not return more times
than the number of encodings the browser supports, so of the remote offer
has more than those, we have an API problem.

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 14:17:04 +0100

if remote offer has more rids than what the browser support then the m line
should be rejected.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 14:22:54 +0100

May be, but without knowing in advance the limit in different
devices/browsers, how many rids do you expect those SFUs will place in the
offer? It would be the "lowest common denominator" which would not satisfy
to anyone.

El lun., 5 nov. 2018 14:17, Sergio Garcia Murillo <
[email-masked]> escribió:

> if remote offer has more rids than what the browser support then the m
> line should be rejected.
>
>
>> As Harlad said before, sender.getParameters() should not return more
>> times than the number of encodings the browser supports, so of the remote
>> offer has more than those, we have an API problem.
>>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 15:24:20 +0100

By the way, according to the simulcast draft that should not be a 
problem at all:

    An answerer that receives an offer with simulcast that lists a number
    of simulcast streams, MAY reduce the number of simulcast streams in
    the answer, but MUST NOT add simulcast streams.

On 05/11/2018 14:22, Iñaki Baz Castillo wrote:
> devices/browsers, how many rids do you expect those SFUs will place in 
> the offer? It would be the "lowest common denominator" which would not 
> satisfy to anyone.
>
> El lun., 5 nov. 2018 14:17, Sergio Garcia Murillo 
> <[email-masked] 
> <mailto:[email-masked]>> escribió:
>
>     if remote offer has more rids than what the browser support then
>     the m line should be rejected.
>
>
>     El lun., 5 nov. 2018 14:14, Iñaki Baz Castillo <[email-masked]
>     <mailto:[email-masked]>> escribió:
>
>         As Harlad said before, sender.getParameters() should not
>         return more times than the number of encodings the browser
>         supports, so of the remote offer has more than those, we have
>         an API problem.
>
>         El lun., 5 nov. 2018 14:03, Sergio Garcia Murillo
>         <[email-masked]
>         <mailto:[email-masked]>> escribió:
>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 15:33:26 +0100

On Mon, 5 Nov 2018 at 15:21, Sergio Garcia Murillo
<[email-masked]> wrote:
>
> By the way, according to the simulcast draft that should not be a problem at all:
>

Not sure if that also covers "recv" simulcast items in the offer.
Probably yes, so ok.

However still useless IMHO. Imagine an SFU that wants to receive, at
least those bitrates (because somehow it knows that all devices can
handle at least 3 simulcast streams):

- 150kbps
- 400kbps
- 1000kbps

Imagine there are also some devices supporting up to 5 streams, so the
SFU produces this simulcast settings in the offer for receiving:

- 150kbps
- 120kbps
- 400kbps
- 750kbps
- 1000kbps

If a device that just supports 3 simulcast streams receives such an
offer, it would discard 2. Which ones? ok, the SDP can include info
about the desired max-bitrate per rid, and we may think: oh, we can
get the list of rids and bitrates  after pc.setRemoteDescription() by
calling pc.getTransceivers()[1].receiver.getParameters().encodings, so
the app can decide which ones to enable.

But that's not true since receiver.getParameters() should not contain
more rids than those supported by the device. Well, this constraint is
really about sender.getParameters(). So you may say: ok we somehow
read the maximum simulcast streams that our browser/device can send
and then choose the appropriate ones from the receiver can call
transceiver.sender.setParameters() with them.. Error! there is no API
for knowing the max number of simulcast streams the browser can
produce.

Too many assumptions and non existing features in this text to make
this scenario really possible.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 18:23:50 +0100

Please, forget my comment about transceiver.receiver.getParameters(),
they are just wrong here.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 5 Nov 2018 15:46:20 -0500

Sergio said:

"here could be a way to overcome this by setting all the send encodings
active flag to false when receiving a simulcast offer. Then app can do a
getParameters, get layer num and rids, configure the settings and enable
the desired ones by setting active to true."

[BA] Yes, this makes sense, and from Nil's note, it appears that this is
the way things work now in Firefox.

On Sun, Nov 4, 2018 at 7:37 PM Sergio Garcia Murillo <
[email-masked]> wrote:

>> > Here is a fiddle which demonstrates that just putting simulcast into
>> the offer, but not calling setEncodings() does not activate simulcast:

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 22:30:32 +0100

On 11/05/2018 02:07 PM, Sergio Garcia Murillo wrote:
> I agree to that. If we are able to find an "easy" (i.e. not
> introducing too many changes) I would not have any issues adding
> support to that.
>
> Something like:
>
> the SFU with default values and all send encodings except first one
> with active=false (this would be the only required change)
>
> encoding parameters ( scaleResolutionDownBy,  maxFramerate,
> maxBitrate, etc) and enables them by setting active=true
>

Yes, I believe so. This is also precisely what I was proposing, except
for the requirement to call addTrack.

Note that our current spec text says that when addTrack is called, a
transceiver is created with the max number of simulcast encodings set to
1, and that this number cannot be increased. This would have to change.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 22:33:34 +0100

> this scenario really possible.

I think the text would have to say that we discard encodings from either
the beginning or the end of the list (I'd suggest the end).
Sorting is going to make some of the people unhappy, some of the time.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 22:35:05 +0100

> 1, and that this number cannot be increased. This would have to change.

Is it not possible to call sender.setParameters(N encodings) after
calling pc.addTrack(track)? why?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 22:37:37 +0100

If in my example above the browser discards encodings at the
beginning, those would remain:

- 400kbps
- 750kbps
- 1000kbps

which are not good for common simulcast usage.

If in my example above the browser discards encodings at the end,
those would remain:

- 150kbps
- 120kbps
- 400kbps

which are not good for common simulcast usage.

:)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Mon, 5 Nov 2018 22:43:57 +0100

> for the requirement to call addTrack.
It would be also possible, to not call addTrack before SLD and use 
replaceTrack on the transceiver once the remote offer is set.
> 1, and that this number cannot be increased. This would have to change.
Yes, it would have to be changed also.

Best regards
Sergio

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 5 Nov 2018 17:16:26 -0500

Harald said:

"Note that our current spec text says that when addTrack is called, a
transceiver is created with the max number of simulcast encodings set to
1, and that this number cannot be increased. This would have to change."

[BA] Most of the code examples I have seen that use addTrack for simulcast
call setParameters prior to createOffer().  To allow this, we might
consider saying that when addTrack returns, the encodings are initially
null, and so can be set by setParameters().

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 23:19:56 +0100

On Mon, 5 Nov 2018 at 23:16, Bernard Aboba <[email-masked]> wrote:

> [BA] Most of the code examples I have seen that use addTrack for simulcast call setParameters prior to createOffer().  To allow this, we might consider saying that when addTrack returns, the encodings are initially null, and so can be set by setParameters().

I'm lost. I've enabled simulcast in Firefox using
sender.setParameters() after both pc.addTrack() and
pc.addTransceiver(), and before pc.createOffer().

So I'm not aware of such a limitation in addTrack that "produces just
1 encoding that cannot be increased". Why not? Why should it work with
addTransceiver() and not with addTrack()?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 5 Nov 2018 17:37:01 -0500

On Mon, Nov 5, 2018 at 5:20 PM Iñaki Baz Castillo <[email-masked]> wrote:

> addTransceiver() and not with addTrack()

[BA] It works today. But currently the specification implies that
setParameters() should fail to set multiple encodings after addTrack. IMHO,
the spec is wrong - it should be allowed at least once before
createOffer(). Should not be too hard to fix.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 5 Nov 2018 23:54:30 +0100

> which are not good for common simulcast usage.

If in your example, the writer of the SDP knew that the browser would
discard encodings at the end, it would send

- 150kbps
- 400kbps
- 1000kbps
- 120kbps
- 750kbps

That's predictable.

If it thought "biggest and smallest are most important", it could also
choose to send

- 120kbps
- 1000kbps
- 400kbps
- 150kbps
- 750kbps

If it thought "most clients are 400kbps, or 1000kbps, the 120kbps client
is only seen occasionally, and handles 3 streams anyway",
it could send

- 400kbps
- 1000kbps
- 120kbps
- 150kbps
- 750kbps

If the behavior of the responder is predictable, the initiator can
achieve the result it wants.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 5 Nov 2018 23:57:35 +0100

> achieve the result it wants.

Right, my fault. In my mind bitrates had to be ordered.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 6 Nov 2018 00:50:30 +0100

On 11/05/2018 11:16 PM, Bernard Aboba wrote:
> [BA] Most of the code examples I have seen that use addTrack for
> simulcast call setParameters prior to createOffer().  To allow this,
> we might consider saying that when addTrack returns, the encodings are
> initially null, and so can be set by setParameters(). 

We could also allow the number of encodings to be changed in general,
with the proviso that if you increase the number, you have to inspect
the result to see how many you got (browser would drop extra elements).

But this is now deep in API-land, and has wandered far from the original
point: That JSEP doesn't allow configuring simulcast from an incoming offer.

I think that if JSEP did not make this statement, it would be OK - since
we would then be able to obey the rules in -simulcast (if the browser
supported it), and leave the rest of the discussion to the API documents.

>
> On Mon, Nov 5, 2018 at 4:30 PM Harald Alvestrand <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     [email-masked] <mailto:[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 6 Nov 2018 00:55:38 +0100

On Tue, 6 Nov 2018 at 00:50, Harald Alvestrand <[email-masked]> wrote:

> But this is now deep in API-land, and has wandered far from the original point: That JSEP doesn't allow configuring simulcast from an incoming offer.
>
> I think that if JSEP did not make this statement, it would be OK - since we would then be able to obey the rules in -simulcast (if the browser supported it), and leave the rest of the discussion to the API documents.

Can I say that it does make any sense that JSEP draft exists? Why
should a separate document *also* specify, for example, the
PeerConnection setRemoteDescription() or addTransceiver() methods?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Tue, 6 Nov 2018 02:02:57 +0100

On 06/11/2018 0:50, Harald Alvestrand wrote:
>> we might consider saying that when addTrack returns, the encodings 
>> are initially null, and so can be set by setParameters().
>

IMHO the number of encodings should be allowed to be changed by 
setParameters (either for a transceiver started via addTrack or 
addTransceiver) until the corresponding m-line is negotiated by an SDP 
O/A (until it has a mid value?). Changing the number of encodings after 
that, I don't think it is allowed by the simulcast draft.

Best regards

Sergio

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Tue, 6 Nov 2018 02:10:14 +0100

changed my mind, as there is a corner case:

addTrack
setParameters with multiple encodings
SRD with a simulcast recv offer with imcompatible rids than the one in the
transceiver

what will happen then?

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 6 Nov 2018 02:21:09 +0100

On Tue, 6 Nov 2018 at 02:10, Sergio Garcia Murillo
<[email-masked]> wrote:
> SRD with a simulcast recv offer with imcompatible rids than the one in the transceiver
>
> what will happen then?

For me this is like if you play by setting different parameters into a
RtpSender before creating an offer (or even before applying it with
sLD).

If you set sending simulcast parameters but do not apply it, and later
you call sRD, then settings generated by sRD should override your
non-applied settings.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 7 Nov 2018 09:14:15 +0700

> On Nov 5, 2018, at 7:14 PM, Iñaki Baz Castillo <[email-masked]> wrote:
> 

Let me put this as politely and straight forward as I can. You are wrong.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 7 Nov 2018 03:55:58 +0100

> Let me put this as politely and straight forward as I can. You are wrong.

May be. But all proposals here about enabling local simulcast upon
receipt of a remote SDP offer require app/JS interaction. The local
app must be able to decide whether send simulcast or not given the
current CPU/battery usage or network conditions. It does not make any
sense that just because the app receives a remote blob, it starts
sending 2-3 video layers without app consent.

That's why, as the discussion has been evolving, the winning proposal
is that in which, upon receipt of the remote SDP offer, the app must
enable simulcast by itself.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 6 Nov 2018 22:09:06 -0500

Cullen said:

"Let me put this as politely and straight forward as I can.  You are wrong."

[BA] I would agree that API interoperability is needed for simulcast, even
if it doesn't yet exist because we're still ironing out the Issues.  The
interest in implementation appears to be quite high, so once we figure
these things out (assuming we make the specification clear enough to avoid
further confusion), we stand a chance of interoperablity in both APIs and
protocols, which is after all, the point of developing a standard.

In that respect, I would prefer if JSEP would include some guidance we
could cite in WebRTC-PC, rather than having to provide clarifications
within the API document itself.

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 7 Nov 2018 07:39:59 +0000

If memory serves, we originally had the model that the author of the
web app and the developer of the receiver of simulcast would cooperate,
in other words the simulcast receiver would send enough info (in form
of whatever) for the web app to correctly set up simulcast using the
available APIs and then send an offer SDP. I think this is what Inaki
is saying.

If this now has changed, and we can have the simulcast receiver send an
SDP offer that is applied in the browser and things work (perhaps with
some web app intervention), and we have people interested in
implementing this, I personally see no reason why that should not be
pursued.

Br, Stefan

On tis, 2018-11-06 at 22:09 -0500, Bernard Aboba wrote:
> [BA] I would agree that API interoperability is needed for simulcast,
> even if it doesn't yet exist because we're still ironing out the
> Issues.  The interest in implementation appears to be quite high, so
> once we figure these things out (assuming we make the specification
> clear enough to avoid further confusion), we stand a chance of
> interoperablity in both APIs and protocols, which is after all, the
> point of developing a standard. 
> 
> In that respect, I would prefer if JSEP would include some guidance
> we could cite in WebRTC-PC, rather than having to provide
> clarifications within the API document itself.
>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 7 Nov 2018 11:36:48 +0100

> available APIs and then send an offer SDP. I think this is what Inaki

This works as long as the initial offer is generated by the browser.

> implementing this, I personally see no reason why that should not be
Yes, there appears to be strong interest.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 7 Nov 2018 12:34:36 +0100

> implementing this, I personally see no reason why that should not be

On of the problems in SDP O/A is the fact that the remote can decide
what the local side should send or stop sending by just sending a blob
to it (a SDP offer or answer), a blob that, due its nature, cannot be
"intercepted" by the JS in the local app to anticipate what it
involves (it's the other way around: first apply whichever unknown
blob you have received from the remote, then realize of what happened
via inspection or via contextless events).

This model is typically argued as a way to get interoperability in
codecs and RTP parameters. But this discussion is going further, since
here we are saying that the remote can send us a blob (that the app
does not know in advance what it means) and enable simulcast in local
side.

In ORTC the remote does NEVER decide what we can send or how we send
it. Assuming capabilities have been exchanged, it's up to the custom
app logic to decide what a peer sends and how it sends that. If you
want to stop receiving the remote video you send a custom JSON to the
server or remote app requesting that. That's friendly. Sending a
secret blob which cannot be managed via JS by the receiver is not
friendly.

So, I want to assume that WebRTC NG will drop SDP and, if so, IMHO it
does not make sense that efforts are spent now in a feature that makes
the dependency on SDP O/A even stronger.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 7 Nov 2018 11:48:41 +0000

On ons, 2018-11-07 at 12:34 +0100, Iñaki Baz Castillo wrote:
> it. 

I will not defend SDP in any way, but I think that even in 1.0 the
remote end has no control on _what_ is sent - it's the local app and
user (and user agent) that controls what input sources to use for track
data, and what tracks to send. 

An non inspected SDP offer/answer can influence _how_ it is sent, but I
think that is per design - no point in encoding with an encoder that
the remote can't decode etc.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 7 Nov 2018 13:00:42 +0100

> the remote can't decode etc.

I can buy that argument when it comes to just that: codec and RTCP
interoperability. Ok, WebRTC 1.0 assumes that app developers cannot
correlate capabilities in JS so the browser does it auto-magically.

But here we are talking about simulcast, which consumes resources and
requires more network uplink. And I want to control what my app sends
(and its max bitrate) because sending 3 streams instead of just 1
means more CPU usage, battery and bandwidth (money).

A received UFO should not change that.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 8 Nov 2018 10:46:11 +0100

> A received UFO should not change that.

Inaki, you're attacking a proposal that has not been made.

We have long ago defined the API that lets the local application control
*exactly* what simulcast can be sent.
What the SDP controls is telling the local browser (and application)
what the other end is willing to accept.