Subject: [rtcweb] When are ICE candidates added to the SDP

From: Eric Rescorla <[email-masked]>
Date: Sun, 11 May 2014 16:51:43 -0700

See: https://github.com/rtcweb-wg/jsep/issues/11

Section 3.4 reads:

When a new ICE candidate is available, the ICE Agent will notify the
application via a callback; these candidates will automatically be
added to the local session description. When all candidates have
been gathered, the callback will also be invoked to signal that the
gathering process is complete.

However, we agreed in London that we would do "always trickle", and
that even if there were candidates available at the time when CreateOffer
(because of candidate pooling) was called, they would not be included
in the initial offer. Consider the following sequence of events.

0. pc = new RTCPeerConnection();
1. pc.AddStream(stream);
2. pc.CreateOffer();
3. CreateOffer callback fires with offer A, and you call
SetLocalDescription();
4. onicecandidate fires with candidate X.
5. pc.CreateOffer()
6. CreateOffer callback fires with offer B.
7. onicecandidate fires with null.

So, in London, I think we agreed that offer A would have no candidates.
The above text implies that if you were to examine localdescription prior
to step #5 or at step #7 it would contain candidate X, and probably that
offer B would
also contain candidate X.

Note that it's quite inconvenient for non-trickle applications to never
have any candidates in the SDP, especially after gathering is completed
at step #7. However it also seems kind of inconsistent to only update the
candidates after SetLocal() has been called.

-Ekr

---

From: "Rauschenbach, Uwe (NSN - DE/Munich)" <[email-masked]>
Date: Mon, 12 May 2014 09:03:34 +0000

Hi Eric, all,

Is the consequence of “always trickle” that ICE candidates have to be added by non-trickling applications via SDP manipulation?
I consider this indeed quite inconvenient for non-trickling applications.

A way out would be that the SDP always contains the candidates collected so far (as currently defined in the JSEP snippet cited),
or that a switch in the API allows an application to configure the behavior when adding ICE candidates.

Kind regards,
Uwe

From: rtcweb [mailto:[email-masked]] On Behalf Of ext Eric Rescorla
Sent: Monday, May 12, 2014 8:52 AM
To: [email-masked]
Subject: [rtcweb] When are ICE candidates added to the SDP

Section 3.4 reads:

3. CreateOffer callback fires with offer A, and you call SetLocalDescription();

So, in London, I think we agreed that offer A would have no candidates.
The above text implies that if you were to examine localdescription prior
to step #5 or at step #7 it would contain candidate X, and probably that offer B would
also contain candidate X.

-Ekr

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 12 May 2014 11:25:23 +0200

2014-05-12 1:51 GMT+02:00 Eric Rescorla <[email-masked]>:
> offer B would also contain candidate X.
>

Well, ugly problem due to ugly design, so ugly solution:

- var myDesc = pc.localDescription;
+ var myDesc = pc.getLocalDescription({candidates: true});

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 12 May 2014 05:39:18 -0700

On Mon, May 12, 2014 at 2:03 AM, Rauschenbach, Uwe (NSN - DE/Munich) <
[email-masked]> wrote:

>  Hi Eric, all,
>
>
>
> Is the consequence of “always trickle” that ICE candidates have to be
> added by non-trickling applications via SDP manipulation?
>

So, my understanding was that we had agreed that no candidates would
be available in the first CreateOffer() no matter how many candidates were
in the pool.

The intent of this message is to ask if that is useful.

-Ekr

 A way out would be that the SDP always contains the candidates collected
> so far (as currently defined in the JSEP snippet cited),
>
> *From:* rtcweb [mailto:[email-masked]] *On Behalf Of *ext Eric
> Rescorla
> *Sent:* Monday, May 12, 2014 8:52 AM
> *To:* [email-masked]
> *Subject:* [rtcweb] When are ICE candidates added to the SDP
>
>
>

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 12 May 2014 15:51:31 +0200

On 05/12/2014 02:39 PM, Eric Rescorla wrote:
>
>
>
> On Mon, May 12, 2014 at 2:03 AM, Rauschenbach, Uwe (NSN - DE/Munich)
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi Eric, all,
>
>      
>
>     Is the consequence of "always trickle" that ICE candidates have to
>     be added by non-trickling applications via SDP manipulation?
>

If an application is written as

1) CreateOffer()
2) wait for the end of candidates callback
3) negotiate

then I would think it might have trouble if all the candidates are
present in the SDP, and therefore no callbacks were called.
Solution: Always fire the end of candidates callback after calling the
success callback, even if all candidates are in the SDP already.

>
> -Ekr
>
>     A way out would be that the SDP always contains the candidates
>     collected so far (as currently defined in the JSEP snippet cited),
>
>     *From:*rtcweb [mailto:[email-masked]
>     <mailto:[email-masked]>] *On Behalf Of *ext Eric Rescorla
>     *To:* [email-masked] <mailto:[email-masked]>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

-- 
Surveillance is pervasive. Go Dark.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 12 May 2014 08:17:13 -0700

On Mon, May 12, 2014 at 6:51 AM, Harald Alvestrand <[email-masked]>wrote:

> then I would think it might have trouble if all the candidates are present
> in the SDP, and therefore no callbacks were called.

So you mean:

1. CreateOffer()
2. success callback fires.
3. SetLocal()
4. onicecandidate(...)
5. onicecandidate(null)

Then later (w/ no changes to the MSTs)

6. CreateOffer()
7. success callback fires
8. SetLocal (????)
9. onicecandidate(null)

I believe this would be a change to the W3C API (not that this is
necessarily bad)

-Ekr

> rtcweb mailing [email-masked]://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
> --

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 12 May 2014 18:14:01 +0200

On 05/12/2014 05:17 PM, Eric Rescorla wrote:
>
>
>
> On Mon, May 12, 2014 at 6:51 AM, Harald Alvestrand 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>>     On Mon, May 12, 2014 at 2:03 AM, Rauschenbach, Uwe (NSN -
>>     DE/Munich) <[email-masked]
>>     <mailto:[email-masked]>> wrote:
>>
>>         Hi Eric, all,
>>
>>         Is the consequence of “always trickle” that ICE candidates
>>         have to be added by non-trickling applications via SDP
>>         manipulation?
>>
>     Solution: Always fire the end of candidates callback after calling
>     the success callback, even if all candidates are in the SDP already.
>
>
> So you mean:
>
> 1. CreateOffer()

Something like that, yes. Or in more detail, for a call requiring 2 
candidates:

1. CreateOffer()
2. success callback fires (1 candidate present)
3. SetLocal()
4. onicecandidate(...)
5. onicecandidate(null)

6. CreateOffer()
7. success callback fires (2 candidates present)
8. SetLocal ()
9. onicecandidate(null)

Then still later (after having done something that requires a 3rd 
candidate):

10. CreateOffer()
11. successCallback fires (2 candidates present)
12. SetLocal()
13. onicecandidate(...)
14. onicecandidate(null)

If we don't have the null onicecandidate in case 2 (step 9), it gets 
hard to support case 3 (step 12-14).

>
> -Ekr
>
>
>>
>>     -Ekr
>>
>>         A way out would be that the SDP always contains the
>>         candidates collected so far (as currently defined in the JSEP
>>         snippet cited),
>>
>>         or that a switch in the API allows an application to
>>         configure the behavior when adding ICE candidates.
>>
>>         Kind regards,
>>         Uwe
>>
>>     [email-masked]  <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Martin Thomson <[email-masked]>
Date: Mon, 12 May 2014 11:59:28 -0700

Why not:

1. pc.createOffer(...)
2. success callback fires (0 candidates)
3. pc.setLocalDescription(offer)
4. onicecandidate fires some number of times
5. onicecandidate fires with a null candidate
6. send pc.localDescription (as opposed to the original offer)

No need to run multiple rounds of createOffer or anything nasty like that.

Note that we don't have an onicegatheringstatechange event that would
let us monitor that state more directly, but the actual code cost is
identical either way.

On 12 May 2014 09:14, Harald Alvestrand <[email-masked]> wrote:
>> <[email-masked]> wrote:
>>>
>>> Hi Eric, all,
>>>
>>>
>>>
> If we don't have the null onicecandidate in case 2 (step 9), it gets hard to
> support case 3 (step 12-14).
>
>
> -Ekr
>
>>
>>
>>
>> -Ekr
>>

---

From: "Rauschenbach, Uwe (NSN - DE/Munich)" <[email-masked]>
Date: Tue, 13 May 2014 03:44:46 +0000

On Mon, May 12, 2014 at 2:03 AM, Rauschenbach, Uwe (NSN - DE/Munich) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Eric, all,

-Ekr

Thanks for clarifying. As long as the candidates are in the SDP that can be obtained via the API after the “end of candidates”, I think non-trickling applications should be OK.
I was concerned they would not end up in the SDP anymore.

-- Uwe

Kind regards,
Uwe

From: rtcweb [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of ext Eric Rescorla
Sent: Monday, May 12, 2014 8:52 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [rtcweb] When are ICE candidates added to the SDP

Section 3.4 reads:

-Ekr

---

From: "Rauschenbach, Uwe (NSN - DE/Munich)" <[email-masked]>
Date: Tue, 13 May 2014 03:46:45 +0000

Hi Martin,

Ok with your proposed flow as long as the SDP obtained in step 6 via the API contains the candidates (currently the W3C spec states they are included when using pc.localDescription).

A question on step 6 and your statement below the sequence: What would be the difference between an SDP created by pc.createOffer in step 6 and one obtained by getting pc.localDescription in step 6? Why do you consider multiple pc.createOffer calls nasty?

Thanks, Uwe 

> -----Original Message-----
> From: rtcweb [mailto:[email-masked]] On Behalf Of ext Martin
> Thomson
> Sent: Tuesday, May 13, 2014 3:59 AM
> To: Harald Alvestrand
> Cc: [email-masked]
> Subject: Re: [rtcweb] When are ICE candidates added to the SDP
> 
> Why not:
> 
> 1. pc.createOffer(...)
> <[email-masked]>
> > wrote:
> present
> the
> collected

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 12 May 2014 21:24:15 -0700

> No need to run multiple rounds of createOffer or anything nasty like that.

This doesn't seem like a complete answer. The spec explicitly allows
multiple CreateOffers, so what is the impact of that?

-Ekr

> > On Mon, May 12, 2014 at 6:51 AM, Harald Alvestrand <[email-masked]
> >
> > wrote:
> >>

---

From: Martin Thomson <[email-masked]>
Date: Mon, 12 May 2014 22:36:07 -0700

On 12 May 2014 20:46, Rauschenbach, Uwe (NSN - DE/Munich)
<[email-masked]> wrote:
>
> A question on step 6 and your statement below the sequence: What would be the difference between an SDP created by pc.createOffer in step 6 and one obtained by getting pc.localDescription in step 6?

The difference is that the second contains candidates.

> Why do you consider multiple pc.createOffer calls nasty?

They aren't convenient.  You risk mistakes.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 12 May 2014 22:37:32 -0700

> multiple CreateOffers, so what is the impact of that?

That would be OK too, except for the async inconvenience associated
with it.  And the risk that you get something different each time (or
that you've messed something up).

---

From: "Rauschenbach, Uwe (NSN - DE/Munich)" <[email-masked]>
Date: Tue, 13 May 2014 06:05:54 +0000

> From: ext Martin Thomson [mailto:[email-masked]]
> 
> > A question on step 6 and your statement below the sequence: What would
> be the difference between an SDP created by pc.createOffer in step 6 and
> one obtained by getting pc.localDescription in step 6?
> 

Uwe: Then, let me elaborate a bit more.

Obviously, we need an SDP reflecting the local configuration for two purposes:
1) To feed it to setLocalDescription in order to start the ICE gathering
2) To send it as an offer to the other peer

The name "createOffer" implies that the SDP delivered as the result of createOffer is the one to use as an offer in negotiation. Looking at the way this mechanism has evolved, this is probably only true anymore if one is prepared to send the offer immediately, and then to trickle ICE candidates. To obtain an SDP to use as an offer in negotiations without trickle ICE, one has to call setLocal, wait for "end of candidates" and then read pc.localDescription.

So "createOffer" in fact does not create an offer, but an initial (or updated) local description that is then fed into setLocalDescription to start the ICE gathering.

Do you agree with this observation? If you do, shouldn't we rename createOffer to something more generic like "createLocalDescription" or similar? "createOffer" does not create the offer (at least it does not do so in case of non-trickling) but the input to setLocal...

---

From: Silvia Pfeiffer <[email-masked]>
Date: Tue, 13 May 2014 17:54:54 +1000

On 13 May 2014 16:06, "Rauschenbach, Uwe (NSN - DE/Munich)" <
[email-masked]> wrote:
> The name "createOffer" implies that the SDP delivered as the result of
createOffer is the one to use as an offer in negotiation. Looking at the
way this mechanism has evolved, this is probably only true anymore if one
is prepared to send the offer immediately, and then to trickle ICE
candidates. To obtain an SDP to use as an offer in negotiations without
trickle ICE, one has to call setLocal, wait for "end of candidates" and
then read pc.localDescription.
>
> So "createOffer" in fact does not create an offer, but an initial (or
updated) local description that is then fed into setLocalDescription to
start the ICE gathering.
>
> Do you agree with this observation? If you do, shouldn't we rename
createOffer to something more generic like "createLocalDescription" or
similar? "createOffer" does not create the offer (at least it does not do
so in case of non-trickling) but the input to setLocal...

I agree, the name is very misleading now.

Silvia.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 13 May 2014 09:45:13 -0700

On 13 May 2014 00:54, Silvia Pfeiffer <[email-masked]> wrote:
>> createOffer is the one to use as an offer in negotiation. Looking at the way
>> this mechanism has evolved, this is probably only true anymore if one is
>> prepared to send the offer immediately, and then to trickle ICE candidates.
>> To obtain an SDP to use as an offer in negotiations without trickle ICE, one
>> has to call setLocal, wait for "end of candidates" and then read
>> pc.localDescription.
>>
>> similar? "createOffer" does not create the offer (at least it does not do so
>> in case of non-trickling) but the input to setLocal...
>>
>

I disagree.  Though I don't find any of this particularly intuitive,
the response to createOffer is still an offer that can be used to
negotiate a session.  The fact that it cannot be used immediately with
some classes of peer is unfortunate, but it's still good for
negotiating with browsers.

So yes, this new createOffer is only good for negotiating with a peer
that supports trickle ICE.  I think that's better than what we had
previously, which is *no idea what the browser was supposed to do*, as
demonstrated by the fact that the only two implementations in
existence did the complete opposite thing.

---

From: Cullen Jennings <[email-masked]>
Date: Sat, 17 May 2014 13:19:04 -0500

On May 11, 2014, at 6:51 PM, Eric Rescorla <[email-masked]> wrote:

> in the initial offer.

Hmm - did we agree they would not be there or they might not be there? 

Either way, I think that things will work better if we can provide whatever information we have available at the time the  call returns. So my preference would be that if they are in the pool, they are returned as early as possible. I can not really see any good reasons for delaying the return of the candidates that are available.  Even when you are doing trickily ICE, this is going to result in less round trips of “trickle” and result in closer synchronization of the connectivity checks from both sides for the candidates that are returned in the first offer. That will help up the success rate of trickle ICE.

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 17 May 2014 17:28:57 -0700

> Hmm - did we agree they would not be there or they might not be there?

My memory is that Justin argued that it would be more consistent to
not have them and I agreed to check to see if that was a big deal
for Firefox. It's not.

> Either way, I think that things will work better if we can provide
> whatever information we have available at the time the  call returns. So my
> preference would be that if they are in the pool, they are returned as
> early as possible.

We're talking about having the onicecandidate callbacks called immediately
on setLocal(), so this should't have much latency impact.

> I can not really see any good reasons for delaying the return of the
> candidates that are available.  Even when you are doing trickily ICE, this
> is going to result in less round trips of “trickle” and result in closer
> synchronization of the connectivity checks from both sides for the
> candidates that are returned in the first offer. That will help up the
> success rate of trickle ICE.
>

See above. I don't think this analysis is really correct.

-Ekr

---

From: Justin Uberti <[email-masked]>
Date: Sun, 18 May 2014 21:14:03 -0700

Agree with Martin's analysis.

Regarding what we should do after a new setLocalDescription, the
implementation does need to change the value of iceGatheringState if new
gathering is going to occur. As pointed out, since we don't have a callback
for iceGatheringState, firing onicecandidate(null) if no gathering needs to
happen seems reasonable, if inelegant.

But the reason for not having an iceGatheringState callback was because we
felt that it was redundant with onicecandidate. That now seems debatable,
and I'd rather add that callback than pound onicecandidate until it looks
like said callback.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 19 May 2014 13:18:17 +0200

On 05/19/2014 06:14 AM, Justin Uberti wrote:
> implementation does need to change the value of iceGatheringState if
> new gathering is going to occur. As pointed out, since we don't have a
> callback for iceGatheringState, firing onicecandidate(null) if no
> gathering needs to happen seems reasonable, if inelegant. 
>
> But the reason for not having an iceGatheringState callback was
> because we felt that it was redundant with onicecandidate. That now
> seems debatable, and I'd rather add that callback than pound
> onicecandidate until it looks like said callback.

The onicecandidate(NULL) always seemed inelegant to me; it wasn't
supplying a candidate, it was indicating something else.

I'd be happy with replacing it with an iceGatheringState change callback
(if we can ensure that this will do the Right Thing in all cases...)

>
>
> On Sat, May 17, 2014 at 5:28 PM, Eric Rescorla <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>
>
>     On Sat, May 17, 2014 at 11:19 AM, Cullen Jennings <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>
>         On May 11, 2014, at 6:51 PM, Eric Rescorla <[email-masked]
>         <mailto:[email-masked]>> wrote:
>
>         Either way, I think that things will work better if we can
>         provide whatever information we have available at the time the
>          call returns. So my preference would be that if they are in
>         the pool, they are returned as early as possible.
>
>
>         I can not really see any good reasons for delaying the return
>         of the candidates that are available.  Even when you are doing
>         trickily ICE, this is going to result in less round trips of
>         "trickle" and result in closer synchronization of the
>         connectivity checks from both sides for the candidates that
>         are returned in the first offer. That will help up the success
>         rate of trickle ICE.
>
>