Subject: [rtcweb] Offer/answer for heterogeneous encode/decode

From: Justin Uberti <[email-masked]>
Date: Fri, 22 Nov 2013 14:11:18 -0800

On Fri, Nov 22, 2013 at 11:57 AM, Paul Giralt <[email-masked]> wrote:

> I may be missing something but I don't know that this is entirely correct.
> From 3264:
>
>    The list of media formats for each media stream conveys two pieces of
>    information, namely the set of formats (codecs and any parameters
>    associated with the codec, in the case of RTP) that the offerer is
>    capable of sending and/or receiving (depending on the direction
>    attributes), and, in the case of RTP, the RTP payload type numbers
>    used to identify those formats.
>
> To me, this means that a capability in an offer or answer (with a
> direction attribute of sendrecv) means that you are capable of both sending
> and receiving for that particular capability. I don't see a way where you
> can specify (within a single m= line) that you are capable of receiving on
> two of the specified payload types, but only capable of sending on one of
> the two. In other words, if an offer or answer specifies both H.264 and VP8
> with a sendrecv attribute, this implies that you are capable of both
> sending and receiving both.
>
> You could, of course, specify two separate m= lines where you specify both
> codecs as recvonly and the codec you support for sending as sendonly, but
> this is very complicated because now you have two separate m= lines for
> what really should be one bi-directional stream.
>
> Someone please correct me if I'm misunderstanding this.
>
>
The sender has full discretion of which codec to actually use. Assuming the
remote side has offered both codec A and codec B (indicating that they are
willing to receive either), the local side can choose to send either A or
B, and can switch at any time.

>From later on in the paragraph from 3264 that you quoted:

   If multiple formats are listed, it
   means that the offerer is capable of making use of any of those
   formats during the session.  In other words, the answerer MAY change
   formats in the middle of the session, making use of any of the
   formats listed, without sending a new offer.

and later:

   In all cases, the formats in the "m=" line MUST be listed in order of
   preference, with the first format listed being preferred.  In this
   case, preferred means that the recipient of the offer SHOULD use the
   format with the highest preference that is acceptable to it.

I certainly think that "is acceptable to it" covers "has an encoder for it".

---

From: "Paul Giralt (pgiralt)" <[email-masked]>
Date: Fri, 22 Nov 2013 23:44:07 +0000

On Nov 22, 2013, at 5:11 PM, Justin Uberti <[email-masked]<mailto:[email-masked]>> wrote:

On Fri, Nov 22, 2013 at 11:57 AM, Paul Giralt <[email-masked]<mailto:[email-masked]>> wrote:
I may be missing something but I don't know that this is entirely correct. >From 3264:

To me, this means that a capability in an offer or answer (with a direction attribute of sendrecv) means that you are capable of both sending and receiving for that particular capability. I don't see a way where you can specify (within a single m= line) that you are capable of receiving on two of the specified payload types, but only capable of sending on one of the two. In other words, if an offer or answer specifies both H.264 and VP8 with a sendrecv attribute, this implies that you are capable of both sending and receiving both.

You could, of course, specify two separate m= lines where you specify both codecs as recvonly and the codec you support for sending as sendonly, but this is very complicated because now you have two separate m= lines for what really should be one bi-directional stream.

The sender has full discretion of which codec to actually use. Assuming the remote side has offered both codec A and codec B (indicating that they are willing to receive either), the local side can choose to send either A or B, and can switch at any time.

and later:

While you’re right that it would work for this scenario, my point was really that Offer/Answer is not really asymmetric as implied earlier. Take for example the hypothetical case where you are only able to decode VP8 but only able to encode H.264. If I offer VP8 as my only codec (because it’s the only thing I’m able to decode therefore I never want anyone to send me anything other than VP8) I cannot send H.264 in the offer because that implies I’m able to decode it. The other side then wants to say it can only receive H.264 so it would have to send back an answer with only H.264. I guess there’s nothing really inherently stopping you from doing this because as far as I can tell, 3264 only says the answer has to be a subset of the offer for multicast streams, however how would the answering side know that the offering side is even capable to receiving H.264? Perhaps Offer/Answer can technically be asymmetric, but it doesn’t seem practical to use it this way because you cannot really indicate your send and receive capabilities independent of each other.

-Paul

---

From: Martin Thomson <[email-masked]>
Date: Fri, 22 Nov 2013 16:06:36 -0800

On 22 November 2013 15:44, Paul Giralt (pgiralt) <[email-masked]> wrote:
> While you’re right that it would work for this scenario, my point was really
> that Offer/Answer is not really asymmetric as implied earlier. Take for
> example the hypothetical case where you are only able to decode VP8 but only
> able to encode H.264. If I offer VP8 as my only codec (because it’s the only
> thing I’m able to decode therefore I never want anyone to send me anything
> other than VP8) I cannot send H.264 in the offer because that implies I’m
> able to decode it. The other side then wants to say it can only receive
> H.264 so it would have to send back an answer with only H.264. I guess
> there’s nothing really inherently stopping you from doing this because as
> far as I can tell, 3264 only says the answer has to be a subset of the offer
> for multicast streams, however how would the answering side know that the
> offering side is even capable to receiving H.264? Perhaps Offer/Answer can
> technically be asymmetric, but it doesn’t seem practical to use it this way
> because you cannot really indicate your send and receive capabilities
> independent of each other.

Judicious application of a=sendonly or a=recvonly avoids this issue.
If you want to send H.264, try a=sendonly on a line with H.264.  If
you want to receive VP8, try a=recvonly on a line with VP8.

---

From: "Paul Giralt (pgiralt)" <[email-masked]>
Date: Sat, 23 Nov 2013 00:20:18 +0000

> you want to receive VP8, try a=recvonly on a line with VP8.

I gave this as an example of a possible way to do it, but that means you need two separate m= lines - one for send and one for receive. Seems strange to do this for something that you really want to be a single bi-directional stream. 

-Paul

---

From: Martin Thomson <[email-masked]>
Date: Fri, 22 Nov 2013 17:03:31 -0800

On 22 November 2013 16:20, Paul Giralt (pgiralt) <[email-masked]> wrote:
> I gave this as an example of a possible way to do it, but that means you need two separate m= lines - one for send and one for receive. Seems strange to do this for something that you really want to be a single bi-directional stream

The bidirectional thing in O/A is the strange way to do things.  Two
lines is pretty intuitive.

---

From: Christer Holmberg <[email-masked]>
Date: Sat, 23 Nov 2013 14:09:15 +0000

Hi,

Like some others, I think the must-decode-both-must-endcode-one alternative as such sounds interesting (whether it has any impact on the licence/IPR issues I'll leave to others to speak for).

However, I have big concerns over the proposal to indicate sendrecv for a codec even if an entity is only able to receive/decode it.

As Justin said, for the "base case" we could probably make it work, as we would know than an rtcweb endpoint must be able to send/encode one of the codecs.

But, what when we add more codecs? Assume Alice sends an offer with H264, VP8 and the new fancy codec X. Bob supports codec X, but he has no clue whether Alice can actually encode it. So, in addition to X Bob will also have to include, and reservce resources for,  H264 and VP8 in the answer - just in case. And, when Alice receives the answer, she can not send a new offer and remove H264 and VP8, because she doesn't know whether Bob can encode X. It will not be possible to negotiate down to a single common codec X - even if both Alice and Bob are able to both encode and decode X.

...and, a legacy answerer which supports X may include only X in the answer, which means Alice has to send a new Offer, without X, in case she can only decode it.

Also, in case there is a gateway/transcoder, even if both endpoints indicate support of the same codec(s), it may still have to insert a transcoder, just in case - as it doesn't know which codecs the endpoints are able to encode.

So, I would STRONLGY RECOMMEND AGAINST allowing to indicate sendrecv for decode-only codecs. I think it would backfire very soon.

Usage of unidirectional, sendonly/recvonly, m- line would of course solve this, and you might know that it is one option we are discussing in CLUE (for other reasons than MTI codec, though).

Another alternative is to perform some kind of capability exchange before the actual offer is sent. But, that would of course increase the number of messages etc.

Regards,

Christer

________________________________
From: rtcweb [[email-masked]] on behalf of Martin Thomson [[email-masked]]
Sent: Saturday, 23 November 2013 3:03 AM
To: Paul Giralt (pgiralt)
Cc: [email-masked]
Subject: Re: [rtcweb] Offer/answer for heterogeneous encode/decode

The bidirectional thing in O/A is the strange way to do things.  Two
lines is pretty intuitive.
_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 24 Nov 2013 17:38:57 +0100

> I gave this as an example of a possible way to do it, but that means you need two separate m= lines - one for send and one for receive. Seems strange to do this for something that you really want t…

An application that can't talk to itself is kind of bizarre too.

I think this is a corner case.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 26 Nov 2013 23:57:00 +0100

On 11/26/2013 11:07 PM, Derek Pang wrote:
> I am new to rtcweb community. Was there any proposal on making both
> Vp8 and H264 decoders MTI in webrtc, but VP8 or H264 encoder to be
> optional ? Does this offer a better option for the marketplace, while
> allowing interoperability.

I appreciate that the amount of reading you have to do in order to catch
up with the archives is enormous, but it is definitely a good thing to
do at this juncture.

This is alternative 12 on http://tools.ietf.org/wg/rtcweb/trac/wiki,
where the current alternatives that the chairs are considering proposing
a vote to choose between have been listed.

>
>
> On Sun, Nov 24, 2013 at 8:38 AM, Harald Alvestrand
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>         On Nov 22, 2013, at 7:06 PM, Martin Thomson
>         <[email-masked] <mailto:[email-masked]>>
>         wrote:
>
>             On 22 November 2013 15:44, Paul Giralt (pgiralt)
>             <[email-masked] <mailto:[email-masked]>> wrote:
>
>         I gave this as an example of a possible way to do it, but that
>         means you need two separate m= lines - one for send and one
>         for receive. Seems strange to do this for something that you
>         really want to be a single bi-directional stream.
>
>
>     [email-masked] <mailto:[email-masked]>

-- 
Surveillance is pervasive. Go Dark.

---

From: Derek Pang <[email-masked]>
Date: Tue, 26 Nov 2013 14:07:18 -0800

I am new to rtcweb community. Was there any proposal on making both Vp8 and
H264 decoders MTI in webrtc, but VP8 or H264 encoder to be optional ? Does
this offer a better option for the marketplace, while allowing
interoperability.

On Sun, Nov 24, 2013 at 8:38 AM, Harald Alvestrand <[email-masked]>wrote:

>> I gave this as an example of a possible way to do it, but that means you
>> need two separate m= lines - one for send and one for receive. Seems
>> strange to do this for something that you really want to be a single
>> bi-directional stream.
>>
>

---

From: Derek Pang <[email-masked]>
Date: Tue, 26 Nov 2013 14:10:56 -0800

-Derek

On Tue, Nov 26, 2013 at 2:07 PM, Derek Pang <[email-masked]> wrote:

> I am new to rtcweb community. Was there any proposal on making both Vp8
> and H264 decoders MTI in webrtc, but VP8 or H264 encoder to be optional ?
> Does this offer a better option for the marketplace, while allowing
> interoperability.
>
>

---

From: Derek Pang <[email-masked]>
Date: Tue, 26 Nov 2013 17:44:01 -0800

Harald,

Thank you for your patience and the pointer.

Much appreciated,
Derek

On Tue, Nov 26, 2013 at 2:57 PM, Harald Alvestrand <[email-masked]>wrote:

> up with the archives is enormous, but it is definitely a good thing to do
> at this juncture.
>
> where the current alternatives that the chairs are considering proposing a
> vote to choose between have been listed.
>
>
>
>
>