Subject: [rtcweb] Asymmetric codecs

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 25 Nov 2013 02:10:37 +0000

I started a draft on how to standardize support for asymmetric codecs back in September. The primary motivation was decoding H.264 and H.265 while only encoding H.264. A secondary motivation, for rtcweb, was decoding H.264 and VP8 while only encoding one of them, as Stefan mentioned. I thought it would be a trivial thing to write up by the Oct. 6 deadline for MTI proposals. The draft is still incomplete and unpublished because all solutions are ugly and overly complex for something so seemingly trivial, just like bundle and unified plan. I now sympathize with Christer and Adam for how they arrived at those (not exactly elegant) solutions, and agree with Martin that a pair of (bundled) m-lines is the most viable (although not exactly elegant) solution for asymmetric codecs.

RFC 4317 drafts originally included asymmetric codecs until they were removed in draft 02:
http://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-mmusic-offer-answer-examples-02.txt

They were removed for fear of issues with shared ports:
http://www.ietf.org/mail-archive/text/mmusic/2003-09.mail

Now that we (almost) have bundle, can we resurrect asymmetric codecs with separate sendonly and recvonly m-lines? Are folks ok with a dependency on bundle? I would be happy to complete a draft in that direction. (That is, a general mmusic draft on asymmetric codecs, not a rtcweb draft on MTI asymmetric codecs. I will move the discussion to mmusic after getting rtcweb opinions first.)

Some may consider this a bizarre corner case. However, for hardware codecs, it is the norm not the exception to have asymmetric encoder and decoder formats. Especially for new formats, which usually appear in hardware decoders well before hardware encoders are available. This happened for H.264, H.265, VP8, and virtually every popular codec format in virtually every popular chipset.

Mo

On 11/23/13, 9:09 AM, Christer Holmberg <[email-masked]> wrote:
Like some others, I think the must-decode-both-must-endcode-one alternative as such sounds interesting (whether it has any impact on the licence/IPR issues I'll leave to others to speak for). However, I have big concerns over the proposal to indicate sendrecv for a codec even if an entity is only able to receive/decode it. … Usage of unidirectional, sendonly/recvonly, m- line would of course solve this, and you might know that it is one option we are discussing in CLUE (for other reasons than MTI codec, though).

On 22 November 2013 16:20, Paul Giralt (pgiralt) <[email-masked]> wrote:
I gave this as an example of a possible way to do it, but that means you need two separate m= lines - one for send and one for receive. Seems strange to do this for something that you really want to be a single bi-directional stream.

On 11/22/13, 8:03 PM, Martin Thomson <[email-masked]> wrote:
The bidirectional thing in O/A is the strange way to do things.  Two lines is pretty intuitive.

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 25 Nov 2013 07:24:00 +0000

Hi,

First, from a pure MTI codec perspective, the question is whether usage of asymmetric codecs would solve the issues that people have.

Second, from an SDP OA/BUNDLE perspective, I don't think we can mandate usage of BUNDLE. And, even if we did, in the initial Offer you still need to assign unique port values for each m- line (as agreed in Vancouver).

...unless, of course, you assign an SDP 'bundle-only' attribute to all m- lines with asymmetric codecs.

Regards,

Christer

From: Mo Zanaty (mzanaty) [mailto:[email-masked]]
Sent: 25. marraskuuta 2013 4:11
To: Christer Holmberg; Martin Thomson; Paul Giralt (pgiralt)
Cc: [email-masked]
Subject: Asymmetric codecs

Mo

On 11/23/13, 9:09 AM, Christer Holmberg <[email-masked]<mailto:[email-masked]>> wrote:
Like some others, I think the must-decode-both-must-endcode-one alternative as such sounds interesting (whether it has any impact on the licence/IPR issues I'll leave to others to speak for). However, I have big concerns over the proposal to indicate sendrecv for a codec even if an entity is only able to receive/decode it. ... Usage of unidirectional, sendonly/recvonly, m- line would of course solve this, and you might know that it is one option we are discussing in CLUE (for other reasons than MTI codec, though).

On 22 November 2013 16:20, Paul Giralt (pgiralt) <[email-masked]<mailto:[email-masked]>> wrote:
I gave this as an example of a possible way to do it, but that means you need two separate m= lines - one for send and one for receive. Seems strange to do this for something that you really want to be a single bi-directional stream.

On 11/22/13, 8:03 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
The bidirectional thing in O/A is the strange way to do things.  Two lines is pretty intuitive.

---

From: cowwoc <[email-masked]>
Date: Mon, 25 Nov 2013 02:34:53 -0500

SDP is supposed to be an implementation detail. Let's figure out whether 
asymmetric codecs solve our problems and wedge it into SDP later.

I suspect that this approach reduces the IPR risk, but H.264 royalties 
still apply, do they not?

Sadly this means we're stuck with: "MUST decode at least two of [VP8, 
H.264, H.261] and MUST encode at least one of them".

Gili

On 25/11/2013 2:24 AM, Christer Holmberg wrote:
>
> Hi,
>
> First, from a pure MTI codec perspective, the question is whether 
> usage of asymmetric codecs would solve the issues that people have.
>
> Second, from an SDP OA/BUNDLE perspective, I don't think we can 
> mandate *usage* of BUNDLE. And, even if we did, in the initial Offer 
> you still need to assign unique port values for each m- line (as 
> agreed in Vancouver).
>
> *From:*Mo Zanaty (mzanaty) [mailto:[email-masked]]
> *Sent:* 25. marraskuuta 2013 4:11
> *To:* Christer Holmberg; Martin Thomson; Paul Giralt (pgiralt)
> *Cc:* [email-masked]
> *Subject:* Asymmetric codecs
>
> I started a draft on how to standardize support for asymmetric codecs 
> back in September. The primary motivation was decoding H.264 and H.265 
> while only encoding H.264. A secondary motivation, for rtcweb, was 
> decoding H.264 and VP8 while only encoding one of them, as Stefan 
> mentioned. I thought it would be a trivial thing to write up by the 
> Oct. 6 deadline for MTI proposals. The draft is still incomplete and 
> unpublished because all solutions are ugly and overly complex for 
> something so seemingly trivial, just like bundle and unified plan. I 
> now sympathize with Christer and Adam for how they arrived at those 
> (not exactly elegant) solutions, and agree with Martin that a pair of 
> (bundled) m-lines is the most viable (although not exactly elegant) 
> solution for asymmetric codecs.
>
> Now that we (almost) have bundle, can we resurrect asymmetric codecs 
> with separate sendonly and recvonly m-lines? Are folks ok with a 
> dependency on bundle? I would be happy to complete a draft in that 
> direction. (That is, a general mmusic draft on asymmetric codecs, not 
> a rtcweb draft on MTI asymmetric codecs. I will move the discussion to 
> mmusic after getting rtcweb opinions first.)
>
> Some may consider this a bizarre corner case. However, for hardware 
> codecs, it is the norm not the exception to have asymmetric encoder 
> and decoder formats. Especially for new formats, which usually appear 
> in hardware decoders well before hardware encoders are available. This 
> happened for H.264, H.265, VP8, and virtually every popular codec 
> format in virtually every popular chipset.
>
> Mo
>
> On 11/23/13, 9:09 AM, Christer Holmberg 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Like some others, I think the must-decode-both-must-endcode-one 
> alternative as such sounds interesting (whether it has any impact on 
> the licence/IPR issues I'll leave to others to speak for). However, I 
> have big concerns over the proposal to indicate sendrecv for a codec 
> even if an entity is only able to receive/decode it. ... Usage of 
> unidirectional, sendonly/recvonly, m- line would of course solve this, 
> and you might know that it is one option we are discussing in CLUE 
> (for other reasons than MTI codec, though).
>
> I gave this as an example of a possible way to do it, but that means 
> you need two separate m= lines - one for send and one for receive. 
> Seems strange to do this for something that you really want to be a 
> single bi-directional stream.
>
> On 11/22/13, 8:03 PM, Martin Thomson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 25 Nov 2013 07:43:51 +0000

Hi,

I'm afraid that SDP has become is a little more than an implementation detail :)

Regards,

Christer

From: rtcweb [mailto:[email-masked]] On Behalf Of cowwoc
Sent: 25. marraskuuta 2013 9:35
To: [email-masked]
Subject: Re: [rtcweb] Asymmetric codecs

SDP is supposed to be an implementation detail. Let's figure out whether asymmetric codecs solve our problems and wedge it into SDP later.

I suspect that this approach reduces the IPR risk, but H.264 royalties still apply, do they not?

Sadly this means we're stuck with: "MUST decode at least two of [VP8, H.264, H.261] and MUST encode at least one of them".

Gili

Regards,

Christer

Cc: [email-masked]<mailto:[email-masked]>

Mo

rtcweb mailing list

[email-masked]<mailto:[email-masked]>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Mon, 25 Nov 2013 14:09:15 +0000

On 2013-11-25 03:11, Mo Zanaty (mzanaty) wrote:
> mentioned. I thought it would be a trivial thing to write up by the Oct.
> 6 deadline for MTI proposals. The draft is still incomplete and
> something so seemingly trivial, just like bundle and unified plan. I now
> sympathize with Christer and Adam for how they arrived at those (not
> exactly elegant) solutions, and agree with Martin that a pair of
> direction. (That is, a general mmusic draft on asymmetric codecs, not a
> rtcweb draft on MTI asymmetric codecs. I will move the discussion to

I've in the past been arguing that the WebRTC API, which is geared 
towards sending tracks, fits best with separate m-lines per direction. 
So I'm all for this (naturally the browser must be able to understand a 
sendrecv m-line if some legacy equipment signals that).

I'm not sure what you mean by the dependency on bundle. Do you mean that 
if bundle is not used the result would be that twice the number of ports 
would be needed for a symmetrical session?

> codecs, it is the norm not the exception to have asymmetric encoder and
> decoder formats. Especially for new formats, which usually appear in
> hardware decoders well before hardware encoders are available. This
> happened for H.264, H.265, VP8, and virtually every popular codec format
> in virtually every popular chipset.
>
> Mo
>
> alternative as such sounds interesting (whether it has any impact on the
> licence/IPR issues I'll leave to others to speak for). However, I have
> big concerns over the proposal to indicate sendrecv for a codec even if
> an entity is only able to receive/decode it. … Usage of unidirectional,
> sendonly/recvonly, m- line would of course solve this, and you might
> know that it is one option we are discussing in CLUE (for other reasons
> than MTI codec, though).
> I gave this as an example of a possible way to do it, but that means you
> need two separate m= lines - one for send and one for receive. Seems
> strange to do this for something that you really want to be a single
> bi-directional stream.
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 25 Nov 2013 16:13:04 +0000

Re: MTI, I don’t think asymmetric codecs will solve all the MTI issues. However, it may solve some practical deployment issues regardless of MTI decisions. If products have a decoder but no encoder, it may help to express that. For example, Chrome can expose its H.264 decoder for webrtc despite no encoder. Hardware products can expose their VP8 decoder for webrtc despite no encoder.

Re: SDP, if you want asymmetric codecs over the same port, is it ok to mandate bundle to achieve this? I started with a single sendrecv m-line with all payload types that can be decoded but not necessarily encoded, as Justin suggested. That would not require bundle. However, it gets ugly when adding extra signaling for which payload types can’t be encoded and multiple O/A rounds for interop with arbitrary peers. The complexity approached bundle, so I ended up with separate sendonly/recvonly m-lines, as Martin suggested.

On 11/25/13, 2:24 AM, Christer Holmberg <[email-masked]<mailto:[email-masked]>> wrote:
Hi,
First, from a pure MTI codec perspective, the question is whether usage of asymmetric codecs would solve the issues that people have.
Second, from an SDP OA/BUNDLE perspective, I don’t think we can mandate usage of BUNDLE. And, even if we did, in the initial Offer you still need to assign unique port values for each m- line (as agreed in Vancouver).
…unless, of course, you assign an SDP ‘bundle-only’ attribute to all m- lines with asymmetric codecs.
Regards,
Christer

Mo

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 25 Nov 2013 16:14:31 +0000

On 11/25/13, 9:09 AM, Stefan Håkansson LK
<[email-masked]> wrote:
I'm not sure what you mean by the dependency on bundle. Do you mean that
if bundle is not used the result would be that twice the number of ports
would be needed for a symmetrical session?

Yes.

---

From: cowwoc <[email-masked]>
Date: Mon, 25 Nov 2013 12:26:30 -0500

<cough> told you so <cough> :)

So folks, what happens when we try removing any mention of SDP from the 
2.0 spec? :) Are you saying this is no longer possible?

Gili

On 25/11/2013 2:43 AM, Christer Holmberg wrote:
> *From:*rtcweb [mailto:[email-masked]] *On Behalf Of *cowwoc
> *Sent:* 25. marraskuuta 2013 9:35
> *To:* [email-masked]
> *Subject:* Re: [rtcweb] Asymmetric codecs
>
>
> SDP is supposed to be an implementation detail. Let's figure out 
> whether asymmetric codecs solve our problems and wedge it into SDP later.
>
>     mandate *usage* of BUNDLE. And, even if we did, in the initial
>     Offer you still need to assign unique port values for each m- line
>     (as agreed in Vancouver).
>
>     *Cc:* [email-masked] <mailto:[email-masked]>
>     I started a draft on how to standardize support for asymmetric
>     codecs back in September. The primary motivation was decoding
>     H.264 and H.265 while only encoding H.264. A secondary motivation,
>     for rtcweb, was decoding H.264 and VP8 while only encoding one of
>     them, as Stefan mentioned. I thought it would be a trivial thing
>     to write up by the Oct. 6 deadline for MTI proposals. The draft is
>     still incomplete and unpublished because all solutions are ugly
>     and overly complex for something so seemingly trivial, just like
>     bundle and unified plan. I now sympathize with Christer and Adam
>     for how they arrived at those (not exactly elegant) solutions, and
>     agree with Martin that a pair of (bundled) m-lines is the most
>     viable (although not exactly elegant) solution for asymmetric codecs.
>
>     Now that we (almost) have bundle, can we resurrect asymmetric
>     codecs with separate sendonly and recvonly m-lines? Are folks ok
>     with a dependency on bundle? I would be happy to complete a draft
>     in that direction. (That is, a general mmusic draft on asymmetric
>     codecs, not a rtcweb draft on MTI asymmetric codecs. I will move
>     the discussion to mmusic after getting rtcweb opinions first.)
>
>     Some may consider this a bizarre corner case. However, for
>     hardware codecs, it is the norm not the exception to have
>     asymmetric encoder and decoder formats. Especially for new
>     formats, which usually appear in hardware decoders well before
>     hardware encoders are available. This happened for H.264, H.265,
>     VP8, and virtually every popular codec format in virtually every
>     popular chipset.
>
>     Mo
>
>     alternative as such sounds interesting (whether it has any impact
>     on the licence/IPR issues I'll leave to others to speak for).
>     However, I have big concerns over the proposal to indicate
>     sendrecv for a codec even if an entity is only able to
>     receive/decode it. ... Usage of unidirectional, sendonly/recvonly,
>     m- line would of course solve this, and you might know that it is
>     one option we are discussing in CLUE (for other reasons than MTI
>     codec, though).
>
>     On 22 November 2013 16:20, Paul Giralt (pgiralt)
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>     I gave this as an example of a possible way to do it, but that
>     means you need two separate m= lines - one for send and one for
>     receive. Seems strange to do this for something that you really
>     want to be a single bi-directional stream.
>
>     [email-masked]  <mailto:[email-masked]>
>

---

From: Basil Mohamed Gohar <[email-masked]>
Date: Mon, 25 Nov 2013 12:53:56 -0500

On 11/25/2013 11:13 AM, Mo Zanaty (mzanaty) wrote:
> Re: MTI, I don’t think asymmetric codecs will solve all the MTI issues.
> However, it may solve some practical deployment issues regardless of MTI
> decisions. If products have a decoder but no encoder, it may help to
> express that. For example, Chrome can expose its H.264 decoder for
> webrtc despite no encoder. Hardware products can expose their VP8
> decoder for webrtc despite no encoder.
> 
> Re: SDP, if you want asymmetric codecs over the same port, is it ok to
> mandate bundle to achieve this? I started with a single sendrecv m-line
> with all payload types that can be decoded but not necessarily encoded,
> as Justin suggested. That would not require bundle. However, it gets
> ugly when adding extra signaling for which payload types can’t be
> encoded and multiple O/A rounds for interop with arbitrary peers. The
> complexity approached bundle, so I ended up with separate
> sendonly/recvonly m-lines, as Martin suggested.

For what it's worth, if it doesn't muddy things up, having *support* for
a different codec being sent from both sides, doesn't sound like a
terribly bad idea.  Maybe a lower-end device, say, a watch or something,
can encode to a simpler codec but decode a more advanced one.  However,
this should be weighed on its own merits, and really shouldn't factor in
at all to the MTI discussion in my opinion.
-- 
Libre Video
http://librevideo.org

---

From: Justin Uberti <[email-masked]>
Date: Mon, 25 Nov 2013 12:21:15 -0800

SDP is just a serialization format, a clumsy yet somehow popular one.

Nothing we decide here on codecs will impose any mandate on what
serialization format is used in WebRTC 2.0.

On Mon, Nov 25, 2013 at 9:26 AM, cowwoc <[email-masked]> wrote:

> I’m afraid that SDP has become is a little more than an implementation
> detail :)
>
>
>
> Regards,
>
>
>
> Christer
>
>
>
> *From:* rtcweb [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *cowwoc
> First, from a pure MTI codec perspective, the question is whether usage of
> asymmetric codecs would solve the issues that people have.
>
>
>
> Second, from an SDP OA/BUNDLE perspective, I don’t think we can mandate
> *usage* of BUNDLE. And, even if we did, in the initial Offer you still
> need to assign unique port values for each m- line (as agreed in Vancouver).
>
>
>
> *From:* Mo Zanaty (mzanaty) [mailto:[email-masked] <[email-masked]>]
>
> I started a draft on how to standardize support for asymmetric codecs back
> in September. The primary motivation was decoding H.264 and H.265 while
> only encoding H.264. A secondary motivation, for rtcweb, was decoding H.264
> and VP8 while only encoding one of them, as Stefan mentioned. I thought it
> would be a trivial thing to write up by the Oct. 6 deadline for MTI
> proposals. The draft is still incomplete and unpublished because all
> solutions are ugly and overly complex for something so seemingly trivial,
> just like bundle and unified plan. I now sympathize with Christer and Adam
> for how they arrived at those (not exactly elegant) solutions, and agree
> with Martin that a pair of (bundled) m-lines is the most viable (although
> not exactly elegant) solution for asymmetric codecs.
>
>
>
> Now that we (almost) have bundle, can we resurrect asymmetric codecs with
> separate sendonly and recvonly m-lines? Are folks ok with a dependency on
> bundle? I would be happy to complete a draft in that direction. (That is, a
> general mmusic draft on asymmetric codecs, not a rtcweb draft on MTI
> asymmetric codecs. I will move the discussion to mmusic after getting
> rtcweb opinions first.)
>
>
>
> happened for H.264, H.265, VP8, and virtually every popular codec format in
> virtually every popular chipset.
>
>
>
> Mo
>
>
>
> licence/IPR issues I'll leave to others to speak for). However, I have big
> concerns over the proposal to indicate sendrecv for a codec even if an
> entity is only able to receive/decode it. … Usage of unidirectional,
> sendonly/recvonly, m- line would of course solve this, and you might know
> that it is one option we are discussing in CLUE (for other reasons than MTI
> codec, though).
>
>
>

---

From: cowwoc <[email-masked]>
Date: Mon, 25 Nov 2013 15:29:43 -0500

Okay, glad to hear it :)

Thanks,
Gili

On 25/11/2013 3:21 PM, Justin Uberti wrote:
> On Mon, Nov 25, 2013 at 9:26 AM, cowwoc <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     So folks, what happens when we try removing any mention of SDP
>     from the 2.0 spec? :) Are you saying this is no longer possible?
>
>     Gili
>
>
>>     I’m afraid that SDP has become is a little more than an
>>     implementation detail :)
>>
>>     Regards,
>>
>>     Christer
>>
>>     *To:* [email-masked] <mailto:[email-masked]>
>>     Sadly this means we're stuck with: "MUST decode at least two of
>>     [VP8, H.264, H.261] and MUST encode at least one of them".
>>
>>     Gili
>>
>>         First, from a pure MTI codec perspective, the question is
>>         whether usage of asymmetric codecs would solve the issues
>>         that people have.
>>
>>         Second, from an SDP OA/BUNDLE perspective, I don’t think we
>>         can mandate *usage* of BUNDLE. And, even if we did, in the
>>         initial Offer you still need to assign unique port values for
>>         each m- line (as agreed in Vancouver).
>>
>>         I started a draft on how to standardize support for
>>         asymmetric codecs back in September. The primary motivation
>>         was decoding H.264 and H.265 while only encoding H.264. A
>>         secondary motivation, for rtcweb, was decoding H.264 and VP8
>>         while only encoding one of them, as Stefan mentioned. I
>>         thought it would be a trivial thing to write up by the Oct. 6
>>         deadline for MTI proposals. The draft is still incomplete and
>>         unpublished because all solutions are ugly and overly complex
>>         for something so seemingly trivial, just like bundle and
>>         unified plan. I now sympathize with Christer and Adam for how
>>         they arrived at those (not exactly elegant) solutions, and
>>         agree with Martin that a pair of (bundled) m-lines is the
>>         most viable (although not exactly elegant) solution for
>>         asymmetric codecs.
>>
>>         codecs with separate sendonly and recvonly m-lines? Are folks
>>         ok with a dependency on bundle? I would be happy to complete
>>         a draft in that direction. (That is, a general mmusic draft
>>         on asymmetric codecs, not a rtcweb draft on MTI asymmetric
>>         codecs. I will move the discussion to mmusic after getting
>>         rtcweb opinions first.)
>>
>>         formats, which usually appear in hardware decoders well
>>         before hardware encoders are available. This happened for
>>         H.264, H.265, VP8, and virtually every popular codec format
>>         Like some others, I think the
>>         must-decode-both-must-endcode-one alternative as such sounds
>>         interesting (whether it has any impact on the licence/IPR
>>         issues I'll leave to others to speak for). However, I have
>>         big concerns over the proposal to indicate sendrecv for a
>>         codec even if an entity is only able to receive/decode it. …
>>         Usage of unidirectional, sendonly/recvonly, m- line would of
>>         course solve this, and you might know that it is one option
>>         we are discussing in CLUE (for other reasons than MTI codec,
>>         though).
>>
>>         I gave this as an example of a possible way to do it, but
>>         that means you need two separate m= lines - one for send and
>>         one for receive. Seems strange to do this for something that
>>         you really want to be a single bi-directional stream.
>>
>>         On 11/22/13, 8:03 PM, Martin Thomson
>>         <[email-masked] <mailto:[email-masked]>>
>>         wrote:
>>