Subject: [payload] VP8 payload, decoder processing capabilities (was Re: [rtcweb] Resolution negotiation - a contribution)

From: Stephan Wenger <[email-masked]>
Date: Mon, 16 Apr 2012 21:40:09 +0000

Hi all,

For context: Harald and myself have been at odds for a while now about the
lack of support for a code point in the VP8 payload that can be used to
negotiate a maximum decoder/bitstream complexity.  Specifically, Harald
(and other VP8 payload folks) suggested that generic mechanisms, such as
the a=framerate attribute of RFC4566 in conjunction with the picture size
aspect of the imageattr of RFC 6236 can be used, at least in the rtcweb
context.  However, as far as I understood our argument, these two
mechanisms in combination are not meant as a limit for decoder complexity
(in terms of samples/sec processing rate), but rather as an indication,
from receiver to sender, of an upper bound of what is "useful to send".
See the email below.  To me, it's quite obvious that an indication of
"useful to send" includes "my decoder can handle this"; however, it could
be more restrictive in that factors other than decoder horsepower could
also be at play, such as screen size, user interface settings, and so on.

I believe that the combination of what can be signaled using the above
mechanisms should be sufficient for rtcweb.  However, I also believe that
it is insufficient for general purpose use, mostly because it requires the
support of RFC 6236, which is not exactly a widely deployed technology.
Further, the a=framerate attribute is not a particularly useful attribute
these days anymore, because variable frame rates, at least for software
encoding/decoding, are the norm.

In previous posts on the payload list (in response to the VP8 payload
WGLC), I have commented on the practical shortcomings of the (lack of)
complexity negotiation, and suggested that this needs to be fixed.

Two options:

1) codify Harald's mechanism (based on a=framerate and imageattr in the
VP8 payload draft, at MUST strength.  "In a declarative context, a
prospective media sender supporting this (VP8 payload) specification MUST
support RFC 4566 a=framerate and RFC6236 imageattr, and MUST include code
points according to both mechanisms to identify the properties of the
media stream.  In an offer-answer context, both offerer and answerer
receiver supporting this VP8 payload specification MUST support
a=framertate and imageattr, and MUST include both in their respective
offer/answer messages, so to identify an operation point that will not
overload the media decoder's capabilities.

The issue with this approach, IMO, is that we are dealing here with three
individual code points (framerate, horizontal and vertical picture size),
where a single code point ought to be sufficient for determining whether a
décor is capable of decoding a stream, at least from a complexity
viewpoint).

2) include, in the V8 payload, a negotiable SDP code point indicating the
complexity of a stream, in units of samples per second processing
requirements or a derivative thereof (such as: levels as used in the MPEG
world).  For example, the VP8 payload could include a single, optional,
negotiable parameter "SamplePerSecond".  If SamplePerSecond were absent in
the SDP, a value of xxxxx must be inferred.  (a sensible value for xxxxx
could be, for example 9216000, which is the number of samples per second
for VGA resolution at 30 Hz).  If SamplePerSecond is present in a
declarative context, it indicates the minimum processing requirements a
decoder must support in order to successfully decode the stream.  In a
symmetric offer-answer context, SamplePerSecond can be used to "dial down"
the complexity of the stream to a value that both encoder and decoder can
support.

My preference is obviously the second proposal, but I'm willing to help
fleshing out either or both of them, just not today :-)

Regards,
Stephan

On 4.13.2012 00:45 , "Harald Alvestrand" <[email-masked]> wrote:

>On 04/12/2012 11:13 PM, Stephan Wenger wrote:
>>
>> On 4.12.2012 12:08 , "Harald Alvestrand"<[email-masked]>  wrote:
>>
>>> On 04/12/2012 08:19 PM, Stephan Wenger wrote:
>>>> Hi Harald,
>>>> Thanks for this strawman.  I believe it should work, but I fail to see
>>>> how
>>>> a two dimensional negotiation requirement (negotiating max values for
>>>> framerate and image size--which, in turn, also has two-dimensional
>>>> properties) leads to better interop than a one dimensional negotiation
>>>> (pixels per second processing requirement).
>>> Stephan,
>>>
>>> I do not see this (or the requirement from the use-cases document)
>>>first
>>> and foremost a decoder complexity negotiation; it is a negotiation of
>>> how much data it is useful to send, given the recipient's intended use
>>> of that data.
>> Then such a negotiation should be executed in addition.  Decoder cycle
>> requirement do matter in practical implementations.
>Feel free to propose language that captures this requirement. As noted,
>my I-D fragment doesn't.
>
>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Apr 2012 02:32:17 +0200

Responding strictly on the payload list, and regarding the VP8 payload 
format only:

The a=framerate attribute is a maximum framerate only. Since the maximum 
framerate is the one that places the most stress on the decoder, the 
ability to use lower framerates is irrelevant to the question of 
restricting the decoder load.

Apart from that, I believe we agree on what we disagree about; the input 
of other participants would be valuable.

                 Harald

On 04/16/2012 11:40 PM, Stephan Wenger wrote:
> On 4.13.2012 00:45 , "Harald Alvestrand"<[email-masked]>  wrote:
>

---

From: "Yuepeiyu (Roy)" <[email-masked]>
Date: Tue, 17 Apr 2012 02:23:42 +0000

Hi Harald and Stephen,

First of all, the "SamplePerSecond" in the option 2 should be the maximal sample per second of the stream, because of the variable frame rates.

Second, if we are talking about signaling of decoder complexity point of view, 
	1) I don't see much difference between option 1 and 2, if option 2 is to signal the real number of "sample per second". 
	2) I believe a derivative of "SamplePerSecond" (still option 2) would make more sense, since to some extent, less values would be seen as a guideline for the encoder to produce the media stream. This is meaningful especially for the declarative context where the encoder has fewer information of the decoder's capability.

Just my 2 cents.
Roy
-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Harald Alvestrand
发送时间: 2012年4月17日 8:32
收件人: Stephan Wenger
抄送: [email-masked]
主题: Re: [payload] VP8 payload, decoder processing capabilities (was Re: [rtcweb] Resolution negotiation - a contribution)

                 Harald

>> Feel free to propose language that captures this requirement. As noted,

_______________________________________________
payload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/payload

---

From: "Roni Even" <[email-masked]>
Date: Tue, 17 Apr 2012 09:48:23 +0300

Hi,
I think that one point that is missing here is if the issue is only for the
receiver capability or request or if there is a need to specify also the
encoder capabilities.
In H.264 (RFC6184) the level parameter signal the encoding and decoding
capability and the answers can signal a lower level (If asymmetry is
supported). The level provides some maximum on what the receiver can
request.
I think that the image attribute does not provide this functionality and I
am not sure if this is the meaning of the SamplesPerSecond parameter.

I am keeping the discussion in payload WG 

Roni Even
As individual

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Stephan Wenger
> Sent: Tuesday, April 17, 2012 12:40 AM
> To: Harald Alvestrand
> Cc: [email-masked]; [email-masked]
> Subject: [payload] VP8 payload, decoder processing capabilities (was
> Re: [rtcweb] Resolution negotiation - a contribution)
> 
> Hi all,
> 
> For context: Harald and myself have been at odds for a while now about
> the lack of support for a code point in the VP8 payload that can be
> used to negotiate a maximum decoder/bitstream complexity.
> Specifically, Harald (and other VP8 payload folks) suggested that
> generic mechanisms, such as the a=framerate attribute of RFC4566 in
> conjunction with the picture size aspect of the imageattr of RFC 6236
> can be used, at least in the rtcweb context.  However, as far as I
> understood our argument, these two mechanisms in combination are not
> meant as a limit for decoder complexity (in terms of samples/sec
> processing rate), but rather as an indication, from receiver to sender,
> of an upper bound of what is "useful to send".
> "useful to send" includes "my decoder can handle this"; however, it
> could be more restrictive in that factors other than decoder horsepower
> could also be at play, such as screen size, user interface settings,
> and so on.
> 
> mechanisms should be sufficient for rtcweb.  However, I also believe
> that it is insufficient for general purpose use, mostly because it
> requires the support of RFC 6236, which is not exactly a widely
> deployed technology.
> Further, the a=framerate attribute is not a particularly useful
> attribute these days anymore, because variable frame rates, at least
> for software encoding/decoding, are the norm.
> 
> prospective media sender supporting this (VP8 payload) specification
> MUST support RFC 4566 a=framerate and RFC6236 imageattr, and MUST
> include code points according to both mechanisms to identify the
> properties of the media stream.  In an offer-answer context, both
> offerer and answerer receiver supporting this VP8 payload specification
> MUST support a=framertate and imageattr, and MUST include both in their
> respective offer/answer messages, so to identify an operation point
> that will not overload the media decoder's capabilities.
> 
> The issue with this approach, IMO, is that we are dealing here with
> three individual code points (framerate, horizontal and vertical
> picture size), where a single code point ought to be sufficient for
> determining whether a décor is capable of decoding a stream, at least
> from a complexity viewpoint).
> 
> 2) include, in the V8 payload, a negotiable SDP code point indicating
> the complexity of a stream, in units of samples per second processing
> requirements or a derivative thereof (such as: levels as used in the
> MPEG world).  For example, the VP8 payload could include a single,
> optional, negotiable parameter "SamplePerSecond".  If SamplePerSecond
> were absent in the SDP, a value of xxxxx must be inferred.  (a sensible
> value for xxxxx could be, for example 9216000, which is the number of
> samples per second for VGA resolution at 30 Hz).  If SamplePerSecond is
> present in a declarative context, it indicates the minimum processing
> requirements a decoder must support in order to successfully decode the
> stream.  In a symmetric offer-answer context, SamplePerSecond can be
> used to "dial down"
> the complexity of the stream to a value that both encoder and decoder
> can support.
> 
> >>>> Thanks for this strawman.  I believe it should work, but I fail to
> >>>> see how a two dimensional negotiation requirement (negotiating max
> >>>> values for framerate and image size--which, in turn, also has
> >>>> two-dimensional
> >>>> properties) leads to better interop than a one dimensional
> >>>> negotiation (pixels per second processing requirement).
> >>> Stephan,
> >>>
> >>>first  and foremost a decoder complexity negotiation; it is a
> >>>negotiation of  how much data it is useful to send, given the
> >>>recipient's intended use  of that data.
> >> Then such a negotiation should be executed in addition.  Decoder
> >> cycle requirement do matter in practical implementations.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Apr 2012 09:31:53 +0200

> am not sure if this is the meaning of the SamplesPerSecond parameter.
This discussion is strictly about the VP8 codec, so the question is 
really whether this functionality is needed for the VP8 codec.

In VP8, there are no options at the encoder, and no intention to add any.
> I am keeping the discussion in payload WG
Respected.

---

From: tom harper <[email-masked]>
Date: Tue, 17 Apr 2012 00:46:56 -0700

I understand where Harald is coming from, VP8 doesn't have the same 
computational jump that h264 has between cavlc and cabac at the 
decoder.   With h264 this is important, as there are a bunch of devices 
that only do the former.  I don't think this applies to vp8 that I know of.

So you don't need a level equivalent.  However, I don't know that the 
standards proposed by Harald do everything either.

Unless I misread, I think rfc 4566 a=framerate is intended to be defined 
at the media level for all video media-  For devices with a general 
frame rate cap this is fine, but often you instead want a codec specific 
max frame rate, due to the varying complexities of differing codecs.  
Also, it isn't clear to me from looking at the imageattr spec that you 
can easily specify a range of resolutions other than listing them out in 
verbose detail.  I don't think that is ideal either.  And I might 
ultimately have a frame rate constraint at a higher resolution that I 
don't have at a lower one.  So that wouldn't be clear using imageattr.

For the cases a=framerate doesn't handle, there still needs to be the 
ability to negotiate a codec specific preferred max display size, as 
well as the codec specific computational limit of decoding,.

So I would vote for a solution like maxmbps- the max complexity is then 
understood.  Then I guess you would need a maxfs equivalent also.  Then 
you could just allow the application to let the frame rate vary as needed.

as me,

Tom

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 17 Apr 2012 11:22:07 -0700

If I understand correctly, one concern is that implementations of VP8 will encounter similar problems to those of H.264 in terms of negotiation of decoder capabilities. H.264 allows you to negotiate max-mbps and max-fs for the maximum macroblocks per second and maximum frame size, respectively. In theory, this should be sufficient. Unfortunately, many real world implementations have limitations in terms of max frame rate as well. While they can receive 1080p at 30 fps, they cannot receive 360p at 60 fps, or 180p at 172 fps. These combinations fall well within the max-mbps and max-fs limits, but they still are not supported due to limitations of implementations.
The proposal of the max-fps parameter for H.264 (http://tools.ietf.org/html/draft-kristensen-avt-rtp-h241param-01) is a response to this. It is a work in progress.
It seems something similar may be necessary in practice for VP8.
I agree with Tom that the existing max-framerate at the media line level is not sufficient.

Cheers,
Charles

> Behalf Of tom harper
> Sent: Tuesday, April 17, 2012 12:47 AM
> To: Roni Even
> Cc: [email-masked]
> Subject: Re: [payload] VP8 payload, decoder processing capabilities
> (was Re: [rtcweb] Resolution negotiation - a contribution)
>

---

From: tom harper <[email-masked]>
Date: Tue, 24 Apr 2012 05:18:49 -0700

I think the max-fps might be what is needed.

A general frame rate display limitation with a device ( i.e. it can only 
do 15, 30 fps) such as a phone could just use frame rate the way Harald 
described on the media line- the case of I can do VGA-30 (but not 120 
fps) and I can do 720p-30.

What this would not handle would be if there is some other overall 
processing cap that is codec specific, and your cpu/hardware could do 
720p 30 and 1080p 15, but 60 is your max due to some programmatic 
limitation or choice.

You could in theory do it with the max-fps  parameter in the following 
manner- Using maxmbps/maxfs, infer the maximum codec frame rate at the 
maximum resolution to be max-mbps/max-fs.  Then you need a max-fps 
parameter to cap your max framerate at lower resolutions.  You could 
infer the max framerate in between by calculating the max-mbps/max-fs 
ratio, assuming it is linear.

You could make max-fps optional by having a default max-fps = 
max-mbps/max-fs.

The above should in theory cover multiple video streams also, where each 
stream's capability could be shaped.

If there is a more complicated use case you would have to probably 
specify exact resolution/frame rate combos, it would be optional, so 
probably doesn't need to be specified here.

Tom

---

From: "Roni Even" <[email-masked]>
Date: Tue, 24 Apr 2012 23:38:08 +0300

Hi,
I would like to explain why we have the maxmbps, maxfs, and all these
parameters for H.264 at least my view.

ITU-T H.264 tables A.1 and A.6 define the level values and as such they also
define the maximum resolution allowed for each level and the maximum decoded
picture buffer size and the maximum video bit rate. Now when an endpoint
claims to support a level it must support all parameters. Now according to
table A.6 if you want to support 1080HD video you must support level 4 which
mean you also support video bit rate of 20Mbit/s and 30 frames per second.
Using only the level parameter you cannot say that you support a lower level
but have larger decoded picture buffer size that will allow you to support
1080HD video at 15fps for example. 
As for the VP8 case note also the need to know the maximum frame size which
is important for the required decoded picture buffer size. This may look not
important for low resolution but when you go to high resolution (even higher
than HD) it may also become a limitation. 
Roni Even
As individual

> Sent: Tuesday, April 24, 2012 3:19 PM
> To: [email-masked]
> A general frame rate display limitation with a device ( i.e. it can
> only do 15, 30 fps) such as a phone could just use frame rate the way
> Harald described on the media line- the case of I can do VGA-30 (but
> not 120
> You could make max-fps optional by having a default max-fps = max-
> mbps/max-fs.
> 
> The above should in theory cover multiple video streams also, where
> each stream's capability could be shaped.
> 
> > If I understand correctly, one concern is that implementations of VP8
> will encounter similar problems to those of H.264 in terms of
> negotiation of decoder capabilities. H.264 allows you to negotiate max-
> mbps and max-fs for the maximum macroblocks per second and maximum
> frame size, respectively. In theory, this should be sufficient.
> Unfortunately, many real world implementations have limitations in
> terms of max frame rate as well. While they can receive 1080p at 30
> fps, they cannot receive 360p at 60 fps, or 180p at 172 fps. These
> combinations fall well within the max-mbps and max-fs limits, but they
> still are not supported due to limitations of implementations.
> > The proposal of the max-fps parameter for H.264
> (http://tools.ietf.org/html/draft-kristensen-avt-rtp-h241param-01) is a
> response to this. It is a work in progress.
> >> Unless I misread, I think rfc 4566 a=framerate is intended to be
> >> defined at the media level for all video media-  For devices with a
> >> general frame rate cap this is fine, but often you instead want a
> >> codec specific max frame rate, due to the varying complexities of
> >> differing codecs.
> >> Also, it isn't clear to me from looking at the imageattr spec that
> >> you can easily specify a range of resolutions other than listing
> them
> >> out in verbose detail.  I don't think that is ideal either.  And I
> >> might ultimately have a frame rate constraint at a higher resolution
> >> that I don't have at a lower one.  So that wouldn't be clear using
> >> imageattr.
> >>
> >> So I would vote for a solution like maxmbps- the max complexity is
> >> then understood.  Then I guess you would need a maxfs equivalent
> >> also.  Then you could just allow the application to let the frame
> >> rate vary as needed.
> >>
> >> as me,
> >>
> >> Tom
> >>
> >>> receiver capability or request or if there is a need to specify
> also
> >> the
> >>> encoder capabilities.
> >>> I think that the image attribute does not provide this
> functionality
> >> and I
> >>>> understood our argument, these two mechanisms in combination are
> >>>> not meant as a limit for decoder complexity (in terms of
> >>>> samples/sec processing rate), but rather as an indication, from
> >>>> receiver to
> >> sender,
> >>>> could also be at play, such as screen size, user interface
> >>>> settings, and so on.
> >>>>
> >>>> mechanisms should be sufficient for rtcweb.  However, I also
> >>>> believe that it is insufficient for general purpose use, mostly
> >>>> because it requires the support of RFC 6236, which is not exactly
> a
> >>>> widely deployed technology.
> >>>> attribute these days anymore, because variable frame rates, at
> >>>> least for software encoding/decoding, are the norm.
> >>>>
> >>>> 1) codify Harald's mechanism (based on a=framerate and imageattr
> in
> >> the
> >>>> prospective media sender supporting this (VP8 payload)
> >>>> specification MUST support RFC 4566 a=framerate and RFC6236
> >>>> imageattr, and MUST include code points according to both
> >>>> mechanisms to identify the properties of the media stream.  In an
> >>>> offer-answer context, both offerer and answerer receiver
> supporting
> >>>> this VP8 payload
> >> specification
> >>>> requirements or a derivative thereof (such as: levels as used in
> >>>> the MPEG world).  For example, the VP8 payload could include a
> >>>> single, optional, negotiable parameter "SamplePerSecond".  If
> >> SamplePerSecond
> >>>> samples per second for VGA resolution at 30 Hz).  If
> >>>> SamplePerSecond
> >> is
> >>>> requirements a decoder must support in order to successfully
> decode
> >> the
> >>>> stream.  In a symmetric offer-answer context, SamplePerSecond can
> >>>> be used to "dial down"
> >>>>>>>> Thanks for this strawman.  I believe it should work, but I
> fail
> >> to

---

From: "Sandy (Alexander) MacInnis" <[email-masked]>
Date: Tue, 24 Apr 2012 21:31:38 +0000

I have another question about this.

Background:
In the VP8 bitstream spec, and in the payload draft, the use of multiple coefficient partitions is optional and not required. 

A VP8 decoder might use multi-threaded processing, such that it can achieve significantly greater throughput by decoding multiple coefficient partitions in parallel. Such a decoder might rely on the presence of a certain number of coefficient partitions in order to achieve a certain level of throughput.

For example, some receiver might comprise a multi-threaded decoder that can decode 1920x1080p60 if the stream has at least four coefficient partitions, but if the stream has only one coefficient partition it might be limited to a significantly smaller product of frame size and frame rate.

Question: With the current draft, how does a multi-threaded decoder know in advance how many coefficient partitions the stream uses, so it can decide whether it can decode a stream of a given frame size and frame rate? Or does this aspect not matter?

--Sandy (Alexander) MacInnis
Broadcom

[...older parts of thread deleted...]

---

From: Olivier Crête <[email-masked]>
Date: Tue, 24 Apr 2012 17:40:58 -0400

> Question: With the current draft, how does a multi-threaded decoder know in advance how many coefficient partitions the stream uses, so it can decide whether it can decode a stream of a given frame…

I think the receiver should include all of that in its SDP, maybe
something like: SamplesPerCoefficientPartition=X
MaxParallelCoeffficientPartitions=Y

That said, I'm concerned that all of this is a really static view of
things, which is useful on a fixed device. But is completely pointless
when writing software that runs on a general purpose computer (such as a
desktop or a smartphone), in which case we want something more dynamic.
So maybe all of this discussion is mostly pointless and we should push
for a more dynamic solution (maybe some kind of RTCP feedback?)

-- 
Olivier Crête
[email-masked]

---

From: "Sandy (Alexander) MacInnis" <[email-masked]>
Date: Wed, 25 Apr 2012 16:26:02 +0000

Thanks Olivier.

I was hoping to hear from someone at Google about this.

BTW, there might be a misunderstanding here. What I have in mind is that a sender would offer a stream indicating how many coefficient partitions that stream uses, along with the frame size and rate (or whatever alternative such as pixel rate or macroblock rate the group decides on). A receiver would be able to decide based on this information whether or not it can decode the stream. The receiver might need at least a certain number of coefficient partitions to be able to decode a stream with a certain frame size & rate. 

Note 1: This does not apply to all receivers, but it might apply to some. 

Note 2: If the stream has more coefficient partitions than the receiver needs, that's not a problem, but if it has too few, that could potentially be a problem. 

In this scenario it doesn't matter whether the receiver's capability are static or dynamic; all that matters is that the receiver knows what its capabilities are at the time it decides whether or not it can decode the stream. 

If the concern is that the receiver's capability might be reduced after it starts to decode the stream, that opens a bigger issue that I was not trying to address. However, I don't think that concern applies to the specific question of how many coefficient partitions the stream has. If the receiver either requires a certain number of coefficient partitions initially, or anticipates that due to dynamic behavior it might require them in the future, it can use that information to decide whether or not to accept the stream. On the other hand, if the receiver might in the future require a smaller frame size & rate stream after the stream has started, that's another matter altogether.