Subject: [rtcweb] New VP8 vs H.264 tests uploaded

From: Adrian Grange <[email-masked]>
Date: Wed, 3 Apr 2013 09:20:03 -0700

We have uploaded a new set of test results comparing VP8 to H.264. This
latest set contains fixes for some of the problems in the previous set. We
would like to extend our thanks to those who made suggestions as to how we
could improve our methodology and encourage suggestions as to how we can
make further improvements.

In these tests we run x264 with the "veryslow" preset and VP8 with the
"good, speed 0" setting in an attempt to produce comparable results.

An overview of our results is available as follows:

- A Quality comparison (psnr):
http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_quality.html

- An Encode Speed comparison:
http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_speed.html

- A comparison of the aggregate time required to decode all of the clips in
the test:
http://downloads.webmproject.org/ietf_tests/vp8vsh264-decodetime.txt

All of our test scripts can either be downloaded from:
http://downloads.webmproject.org/ietf_tests/vp8_vs_h264.tar.xz
or checked out of our git/gerrit repository:
git clone http://git.chromium.org/webm/vpx_codec_comparison.git

The file README.txt, contained within, presents details of how to build and
run the tests.

The compressed video files--the output from the quality tests--can also be
downloaded:

VP8:
http://downloads.webmproject.org/ietf_tests/vp8_videos/index.html

H.264:
http://downloads.webmproject.org/ietf_tests/h264_videos/index.html

Adrian Grange

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 04 Apr 2013 10:53:53 +0200

Hi Adrian,

Could you explain how the encoding parametrization is comparable?

x264 --nal-hrd cbr --vbv-maxrate ${rate} --vbv-bufsize ${rate} \
       --vbv-init 0.8 --bitrate ${rate} --fps ${frame_rate} \
       --profile baseline --no-scenecut --keyint infinite --preset 
veryslow \
       --input-res ${width}x${height} \
       --tune psnr \
       -o ./encoded_clips/h264/${clip_stem}_${rate}kbps.mkv ${filename} \
       2> ./logs/h264/${clip_stem}_${rate}kbps.txt

vs:

  ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate} 
--kf-min-dist=3000 \
       --kf-max-dist=3000 --cpu-used=0 --fps=${frame_rate}/1 
--static-thresh=0 \
       --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 
--max-q=56 \
       --undershoot-pct=100 --overshoot-pct=15 --buf-sz=1000 \
       --buf-initial-sz=800 --buf-optimal-sz=1000 --max-intra-rate=1200 \
       --resize-allowed=0 --drop-frame=0 --passes=1 --good 
--noise-sensitivity=0 \
       -w ${width} -h ${height} ${filename} --codec=vp8 \
       -o ./encoded_clips/vp8/${clip_stem}_${rate}kbps.webm \
       &>./logs/vp8/${clip_stem}_${rate}kbps.txt

Best regards
Sergio

El 03/04/2013 18:20, Adrian Grange escribió:
> We have uploaded a new set of test results comparing VP8 to H.264. 
> This latest set contains fixes for some of the problems in the 
> previous set. We would like to extend our thanks to those who made 
> suggestions as to how we could improve our methodology and encourage 
> suggestions as to how we can make further improvements.
>
> - A comparison of the aggregate time required to decode all of the 
> clips in the test: 
> The file README.txt, contained within, presents details of how to 
> build and run the tests.
>
> The compressed video files--the output from the quality tests--can 
> also be downloaded:
>
> VP8:
> http://downloads.webmproject.org/ietf_tests/vp8_videos 
> <http://downloads.webmproject.org/ietf_tests/vp8_videos/>/index.html
>
> H.264:
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 04 Apr 2013 15:30:47 +0200

>       &>./logs/vp8/${clip_stem}_${rate}kbps.txt

Both have the same target bitrate and the same resolution, and neither 
generates periodic keyframes.

Apart from that, I think they are comparable by virtue of being the 
parameters that were recommended for this test by people who like this 
particular codec implementation.

Was there any specific parameter or parameter set you were wondering about?

>> In these tests we run x264 with the "veryslow" preset and VP8 with 
>> the "good, speed 0" setting in an attempt to produce comparable results.
>>

---

From: Luca De Cicco <[email-masked]>
Date: Thu, 4 Apr 2013 15:59:48 +0200

Why did you use baseline instead of mainline or highline for x264? It
should be better to compare
vp8 at least with the mainline profile.

Cheers,
Luca

On Thu, Apr 4, 2013 at 3:30 PM, Harald Alvestrand <[email-masked]> wrote:
>       --profile baseline --no-scenecut --keyint infinite --preset veryslow \
>  ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate} --kf-min-dist=3000
> \
>       --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 --max-q=56 \

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 04 Apr 2013 16:30:46 +0200

> vp8 at least with the mainline profile.
Better in what sense?

This is the discussion of the MTI for RTCWEB; the proponents of H.264 as 
MTI have proposed:

    The core of the proposal is that H.264 Constrained Baseline Profile
    Level 1.2 MUST be supported as Mandatory To Implement video codec.
    To enable higher quality for devices capable of it, support for H.264
    Constrained High Profile Level 1.3, extended to support 720p
    resolution at 30 Hz framerate is RECOMMENDED.

(draft-burman-rtcweb-h264-proposal-01.txt section 1)

The scripts we have published is trying to compare what the H.264 
proponents have proposed for MTI with what the VP8 proponents have 
proposed for MTI.

---

From: Luca De Cicco <[email-masked]>
Date: Thu, 4 Apr 2013 16:38:25 +0200

Ok, thanks for very clear explanation :).

Luca

On Thu, Apr 4, 2013 at 4:30 PM, Harald Alvestrand <[email-masked]> wrote:
> This is the discussion of the MTI for RTCWEB; the proponents of H.264 as MTI
> have proposed:
>
> The scripts we have published is trying to compare what the H.264 proponents
> have proposed for MTI with what the VP8 proponents have proposed for MTI.
>
>
>>
>> Cheers,
>> Luca
>>
>>> --kf-min-dist=3000

---

From: Thomas Davies <[email-masked]>
Date: Thu, 04 Apr 2013 16:05:06 +0100

Harald,

I think there are quite a few problems with the comparison you have posted.

1. Looking at the sequences there is a very major difference between the 
initial intra frame qualities. When I encode just one frame of sequence 
gipsrecomotion using the parameters in the script at 1Mb/s then the 
intra frame is 3 times larger with vp8 than with x264.

With video conferencing content, the quality of the initial I frame has 
a big impact that can last for many seconds - certainly the length of 
these clips. You can easily get gains by increasing the quality 
difference between an I frame and subsequent frames.

x264 seems to have a policy of initially undershooting the bitrate 
substantially and ramping up, whereas vpxenc has a different approach. 
During this ramp up period the quality is very much worse. I can't find 
a way to persuade x264 to behave differently.

This is a good illustration of why including rate control in comparisons 
is a bad idea.

2. Likewise, looking at the individual frame sizes, it seems vpxenc is 
using a quality hierarchy with a length of 8 ("hiercharchical-P") where 
every 8th frame is about 4x bigger than the others. x264 has a constant 
target per frame.

Hierarchical P frames are a really good idea, and can easily get you 
10-20% gain with a big separation like this, at a cost in latency. Again 
I don't know how to make x264 do this, but the technique is applicable 
to any codec and is used in the JM reference.

3. The x264 settings are a bit of a black art, but appear not to be 
ideal after all. I am definitely no expert but I found that when 
encoding gipsrecomotion at 1Mb/s:

- setting --threads 1 improves quality by a full 1dB (vpxenc seems to 
run single threaded by default)
- reducing the number of references from 3 to 2 (--ref 2) reduces the 
load very substantially at very little loss (0.2dB or so).

So with --threads 1 --ref 2, I found x264 ran more than 2x faster than 
vpxenc for this data point and had much better quality than before. 
vpxenc is still better (about 1dB), but very possibly within the range 
of hierarchical P coding improvements.

Incidentally, I don't think that x264 performs particularly well at 
these high complexity settings, at least for video conferencing, no 
doubt as other more practical settings have been targeted. x264 appears 
to have a quality ceiling that the JM does not have.

4. Another (smaller) issue is that the reported PSNR is combined luma 
and chroma over all frames. It's relatively easy to improve chroma PSNR 
at a small cost in bits, and usually it is best to ignore chroma PSNR or 
(possibly) give it a small weight. The arithmetic mean of frame PSNRs is 
generally used rather than the PSNR of the whole sequence, also. I would 
very much like separate component PSNRs in tests. The figures I quote 
above are luma PSNR.

If the purpose is to show whether vp8 is superior as a *technology* to 
h264 CBP, then I think the comparison should use the best settings you 
have (ideally with a special full-on non-real time implementation) and 
test against the JM reference encoder. Ideally you would use the same or 
similar GOP structures, number of references, prediction and QP 
hierarchies.

Comparing different real-time implementations of different codecs trying 
to do high quality coding with different GOP structures and using rate 
control with different strategies is just a waste of time. The first two 
elements in the list above are alone worth a very significant amount of 
bit rate.

On the other hand, a quick perusal of the actual tools would suggest 
that vp8 and h264 CBP are likely "comparable" and the variation between 
implementations of the same technology would be bigger than the 
variation between the technologies. If we could agree that then a lot of 
time could be saved.

best regards

Thomas

---

From: Matthew Kaufman <[email-masked]>
Date: Thu, 04 Apr 2013 08:08:44 -0700

> We have uploaded a new set of test results comparing VP8 to H.264.

Why?

Is there really anyone on the list who is making a decision based on 
perceived quality?

And if so, would they care what anyone's tests other than their own showed?

Matthew Kaufman

---

From: Adam Roach <[email-masked]>
Date: Thu, 04 Apr 2013 10:23:19 -0500

> And if so, would they care what anyone's tests other than their own 

I for one, applaud the approach of publishing the exact tools and 
options used for encoding. Without that kind of transparency, it becomes 
very difficult to trust purported output. It's a "many eyes" approach to 
finding flaws with the comparisons thet goes a long way towards 
preventing claims of tester bias.

I do find it a bit odd that, although we've had proponents of both 
solutions show what they purport to be output of the codecs, the only 
"don't take my word for it, replicate my results at home on your own 
machine" transparency I've seen has been from the VP8 camp.

/a

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Thu, 04 Apr 2013 09:14:42 -0700

On 04/04/2013 08:23 AM, Adam Roach wrote:
> I for one, applaud the approach of publishing the exact tools and options
> used for encoding. Without that kind of transparency, it becomes very
> difficult to trust purported output. It's a "many eyes" approach to finding
> flaws with the comparisons thet goes a long way towards preventing claims
> of tester bias.
> 

That shows that there is a fundamental asymmetry between the VP8 camp and the
H.264.  It is nice to see technical comparisons like this one - and more of
them are needed - but a more important discussion, in my opinion, should be
about the fact that the VP8 camp is fundamentally about inclusiveness and the
H.264 camp about exclusiveness.  Cullen's comment in the meeting about the
fact that there is only 4 browsers and that they all but Mozilla have a fully
paid license for H.264 is symptomatic of the blindness of the H.264 camp to
the need of the RTCWeb/Webrtc community at large.  OTOH, the VP8 camp
consistently demonstrated, as with the OP email, willingness to work with the
community and make RTCWeb/Webrtc available and affordable to the whole
spectrum of participants.

As an independent developer, I do not need to look at the results of the tests
to choose my camp.

- -- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: http://blog.marc.petit-huguenin.org
Profile: http://www.linkedin.com/in/petithug

---

From: Leon Geyser <[email-masked]>
Date: Thu, 4 Apr 2013 18:55:30 +0200

> h264 CBP, then I think the comparison should use the best settings you have
> (ideally with a special full-on non-real time implementation) and test
> against the JM reference encoder. Ideally you would use the same or similar
> GOP structures, number of references, prediction and QP hierarchies.
>
I thought WebRTC was meant for real-time communication. What would it
benefit us if we test settings that won't be used or can't be used in
practice?

The tests need to test the encoders at realtime/low latency and at a
constrained bitrate mode like CBR. We aren't archiving videos here :)

A graph that shows the bitrate over time for each clip could be usefull to
make sure that no encoder spikes the bitrate too high at certain moments.
I welcome changes to the encoder settings as long as they stay realtime/low
latency and constrained bitrate.

On 4 April 2013 17:05, Thomas Davies <[email-masked]> wrote:

> gipsrecomotion using the parameters in the script at 1Mb/s then the intra
> frame is 3 times larger with vp8 than with x264.
>
> With video conferencing content, the quality of the initial I frame has a
> big impact that can last for many seconds - certainly the length of these
> clips. You can easily get gains by increasing the quality difference
> between an I frame and subsequent frames.
>
> During this ramp up period the quality is very much worse. I can't find a
> way to persuade x264 to behave differently.
>
> 10-20% gain with a big separation like this, at a cost in latency. Again I
> don't know how to make x264 do this, but the technique is applicable to any
> codec and is used in the JM reference.
>
> 3. The x264 settings are a bit of a black art, but appear not to be ideal
> after all. I am definitely no expert but I found that when encoding
> gipsrecomotion at 1Mb/s:
>
> - setting --threads 1 improves quality by a full 1dB (vpxenc seems to run
> single threaded by default)
> - reducing the number of references from 3 to 2 (--ref 2) reduces the load
> very substantially at very little loss (0.2dB or so).
>
> vpxenc for this data point and had much better quality than before. vpxenc
> is still better (about 1dB), but very possibly within the range of
> hierarchical P coding improvements.
>
> Incidentally, I don't think that x264 performs particularly well at these
> high complexity settings, at least for video conferencing, no doubt as
> other more practical settings have been targeted. x264 appears to have a
> quality ceiling that the JM does not have.
>
> 4. Another (smaller) issue is that the reported PSNR is combined luma and
> chroma over all frames. It's relatively easy to improve chroma PSNR at a
> small cost in bits, and usually it is best to ignore chroma PSNR or
> very much like separate component PSNRs in tests. The figures I quote above
> are luma PSNR.
>
> h264 CBP, then I think the comparison should use the best settings you have
> (ideally with a special full-on non-real time implementation) and test
> against the JM reference encoder. Ideally you would use the same or similar
> GOP structures, number of references, prediction and QP hierarchies.
>
> elements in the list above are alone worth a very significant amount of bit
> rate.
>
> On the other hand, a quick perusal of the actual tools would suggest that
> vp8 and h264 CBP are likely "comparable" and the variation between
> implementations of the same technology would be bigger than the variation
> between the technologies. If we could agree that then a lot of time could
> be saved.
>
> best regards
>
> Thomas
>
>
>
>
>  - A comparison of the aggregate time required to decode all of the clips
> in the test:
>  The file README.txt, contained within, presents details of how to build
> and run the tests.
>
>  The compressed video files--the output from the quality tests--can also
> be downloaded:
>
>  VP8:
> rtcweb mailing [email-masked]://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
>
> rtcweb mailing [email-masked]://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
>

---

From: "Roy, Radhika R CIV USARMY (US)" <[email-masked]>
Date: Thu, 4 Apr 2013 17:02:43 +0000

Classification: UNCLASSIFIED
Caveats: NONE

Yes, real-time applications (i.e. two-way or multipoint conversation)
performance requirements are fundamentally much more stringent than those of
the near-real-time (i.e. video streaming)/nor-real-time applications.

So, settings for testing for each one of those applications must be done
accordingly.

Best regards,
Radhika

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Leon Geyser
Sent: Thursday, April 04, 2013 12:56 PM
To: Thomas Davies
Cc: [email-masked]
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded

	If the purpose is to show whether vp8 is superior as a *technology*
to h264 CBP, then I think the comparison should use the best settings you
have (ideally with a special full-on non-real time implementation) and test
against the JM reference encoder. Ideally you would use the same or similar
GOP structures, number of references, prediction and QP hierarchies. 

 

	Harald,

Classification: UNCLASSIFIED
Caveats: NONE

---

From: "Thomas Davies (thdavies)" <[email-masked]>
Date: Thu, 4 Apr 2013 18:23:58 +0000

The single-core encode speed I get from these settings is <5fps for the example I chose (which is one of the data points), so these are *not* real time tests.

My understanding is that Google have selected a non-real-time encoding mode to show the capabilities of the VP8 technology and attempted to do the same with x264, hence the --tune veryslow parameter. 

Best regards

Thomas

-----Original Message-----
From: Roy, Radhika R CIV USARMY (US) [mailto:[email-masked]] 
Sent: 04 April 2013 18:03
To: Leon Geyser; Thomas Davies (thdavies)
Cc: [email-masked]
Subject: RE: [rtcweb] New VP8 vs H.264 tests uploaded (UNCLASSIFIED)

Best regards,
Radhika

-----Original Message-----

 

	Harald,

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 04 Apr 2013 20:39:17 +0200

> My understanding is that Google have selected a non-real-time encoding mode to show the capabilities of the VP8 technology and attempted to do the same with x264, hence the --tune veryslow paramete…

One of the tests we provide is a comparision of the encode time vs 
quality for the two codecs.
Of course this one varies a LOT even between different developers' 
machines (the absolute numbers, that is) - machines are very different 
in how fast they encode. Try it out!

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 04 Apr 2013 21:34:06 +0200

On 04/04/2013 05:05 PM, Thomas Davies wrote:
> 1. Looking at the sequences there is a very major difference between 
> the initial intra frame qualities. When I encode just one frame of 
> sequence gipsrecomotion using the parameters in the script at 1Mb/s 
> then the intra frame is 3 times larger with vp8 than with x264.
>
> With video conferencing content, the quality of the initial I frame 
> has a big impact that can last for many seconds - certainly the length 
> of these clips. You can easily get gains by increasing the quality 
> During this ramp up period the quality is very much worse. I can't 
> find a way to persuade x264 to behave differently.
>
> This is a good illustration of why including rate control in 
> comparisons is a bad idea.

Alternatively, it is a good illustration of why demonstrating codecs 
without rate control when the applications they are going to be used in 
require rate control is a thoroughly bogus idea.

A large size of the initial iframe doesn't necessarily mean that the 
bitrate is very high for the first 0.05 seconds, however - one might 
choose to accept a few extra milliseconds of "setup delay" while the 
first iframe is transmitted rather than going with a lower quality 
experience at the start of the call.

> using a quality hierarchy with a length of 8 ("hiercharchical-P") 
> where every 8th frame is about 4x bigger than the others. x264 has a 
> constant target per frame.
>
> 10-20% gain with a big separation like this, at a cost in latency. 
> Again I don't know how to make x264 do this, but the technique is 
> applicable to any codec and is used in the JM reference.

I'm sure you can plug in any encoder you want into the test framework if 
you want to. I'm not sure we can plug in the JM reference codec, however 
- I don't remember offhand what license it's under, but suspect that 
it's "only for use within the MPEG standards process".

> doubt as other more practical settings have been targeted. x264 
> appears to have a quality ceiling that the JM does not have.

Thanks - certainly worth looking at!

> and chroma over all frames. It's relatively easy to improve chroma 
> PSNR at a small cost in bits, and usually it is best to ignore chroma 
> PSNR or (possibly) give it a small weight. The arithmetic mean of 
> frame PSNRs is generally used rather than the PSNR of the whole 
> sequence, also. I would very much like separate component PSNRs in 
> tests. The figures I quote above are luma PSNR.

"is generally used" is such a nice statement ... the whole psnr.c source 
code in the test framework is 148 lines long, most of it error checking; 
I'm happy to see a suggestion for a different psnr measurement method - 
preferably with accompanying source (and a license that allows us to use 
it).

> test against the JM reference encoder. Ideally you would use the same 
> or similar GOP structures, number of references, prediction and QP 
> hierarchies.

Why? I've always had a problem with this idea that if one technology 
does better than another because it's good at using some particular 
tool, that means that one should handicap it by preventing it from using 
that particular tool in the way it's good at.

Test conditions should (in my opinion) be limited to what's observable, 
and what matters to the particular conditions we're looking at. In the 
realtime case, delay, ability to stay within reasonable bandwidth bounds 
and picture quality are the most obvious things that come to mind; at 
other times, resilience and adaptability have also been mentioned. But 
limiting allowable QP hierarchies is, to my mind, not only irrelevant, 
it's absurd.

>
> Comparing different real-time implementations of different codecs 
> trying to do high quality coding with different GOP structures and 
> using rate control with different strategies is just a waste of time. 
> The first two elements in the list above are alone worth a very 
> significant amount of bit rate.
>
> that vp8 and h264 CBP are likely "comparable" and the variation 
> between implementations of the same technology would be bigger than 
> the variation between the technologies. If we could agree that then a 
> lot of time could be saved.

Indeed. Of course, the presentation by Cisco in Orlando did not make it 
easy to see that such an agreement was going to be achieved.

Are you suggesting that this is a reasonable conclusion?

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 4 Apr 2013 23:28:32 +0000

Realtime/low latency and constrained bitrate are obviously important for the actual implementation used. Thomas was pointing out that these factors have nothing to do with the codec technology itself, since they are purely encoder implementation optimizations. There is nothing in the VP8 or H.264 standard that uniquely provides realtime/low latency or constrained bitrate. Those are attributes of encoder implementations which are not part of the standard.

So the question was whether we care about evaluating codec technology or specific implementations. If the former, then tests should be staged in the same way codec experts evaluate codec technology/tools. If the latter, then tests should be staged using the target implementations.

I'm not aware of conferencing applications which use x264, because it was designed and optimized for transcoding (dvd rips to blu-ray) not conferencing. Most importantly, x264 cbr mode is inappropriate for conferencing since it is for broadcast MPEG transport streams that must be absolutely CBR to avoid M2TS-mux overflow or underflow, and it will actually insert filler data instead of real frame data to hit the CBR rate exactly. Looking at the results which show the worst H.264 bitrate (62% above VP8) in gipsrecstat_1280_720_50_1485kbps.mkv, there is almost as much filler data as real frame data, meaning the true bitrate of real frame data is almost half what is reported in the results. (See attached if it makes it through.)

While the results are bad, the methodology, effort and transparency are very good (if we want to compare implementations not standards). I can rerun without the bogus fillers and post the results next week, unless someone else can do it faster. But as Thomas pointed out, the technologies themselves are comparable as far as coding tools, so any results which show significant differences are either suspect or explained by differences in encoder implementations or settings not the codec technology itself.

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Leon Geyser
Sent: Thursday, April 04, 2013 12:56 PM
To: Thomas Davies (thdavies)
Cc: [email-masked]
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded

If the purpose is to show whether vp8 is superior as a *technology* to h264 CBP, then I think the comparison should use the best settings you have (ideally with a special full-on non-real time implementation) and test against the JM reference encoder. Ideally you would use the same or similar GOP structures, number of references, prediction and QP hierarchies.
I thought WebRTC was meant for real-time communication. What would it benefit us if we test settings that won't be used or can't be used in practice?

The tests need to test the encoders at realtime/low latency and at a constrained bitrate mode like CBR. We aren't archiving videos here :)

A graph that shows the bitrate over time for each clip could be usefull to make sure that no encoder spikes the bitrate too high at certain moments.
I welcome changes to the encoder settings as long as they stay realtime/low latency and constrained bitrate.

On 4 April 2013 17:05, Thomas Davies <[email-masked]<mailto:[email-masked]>> wrote:
Harald,

1. Looking at the sequences there is a very major difference between the initial intra frame qualities. When I encode just one frame of sequence gipsrecomotion using the parameters in the script at 1Mb/s then the intra frame is 3 times larger with vp8 than with x264.

With video conferencing content, the quality of the initial I frame has a big impact that can last for many seconds - certainly the length of these clips. You can easily get gains by increasing the quality difference between an I frame and subsequent frames.

x264 seems to have a policy of initially undershooting the bitrate substantially and ramping up, whereas vpxenc has a different approach. During this ramp up period the quality is very much worse. I can't find a way to persuade x264 to behave differently.

This is a good illustration of why including rate control in comparisons is a bad idea.

2. Likewise, looking at the individual frame sizes, it seems vpxenc is using a quality hierarchy with a length of 8 ("hiercharchical-P") where every 8th frame is about 4x bigger than the others. x264 has a constant target per frame.

Hierarchical P frames are a really good idea, and can easily get you 10-20% gain with a big separation like this, at a cost in latency. Again I don't know how to make x264 do this, but the technique is applicable to any codec and is used in the JM reference.

3. The x264 settings are a bit of a black art, but appear not to be ideal after all. I am definitely no expert but I found that when encoding gipsrecomotion at 1Mb/s:

- setting --threads 1 improves quality by a full 1dB (vpxenc seems to run single threaded by default)
- reducing the number of references from 3 to 2 (--ref 2) reduces the load very substantially at very little loss (0.2dB or so).

So with --threads 1 --ref 2, I found x264 ran more than 2x faster than vpxenc for this data point and had much better quality than before. vpxenc is still better (about 1dB), but very possibly within the range of hierarchical P coding improvements.

Incidentally, I don't think that x264 performs particularly well at these high complexity settings, at least for video conferencing, no doubt as other more practical settings have been targeted. x264 appears to have a quality ceiling that the JM does not have.

4. Another (smaller) issue is that the reported PSNR is combined luma and chroma over all frames. It's relatively easy to improve chroma PSNR at a small cost in bits, and usually it is best to ignore chroma PSNR or (possibly) give it a small weight. The arithmetic mean of frame PSNRs is generally used rather than the PSNR of the whole sequence, also. I would very much like separate component PSNRs in tests. The figures I quote above are luma PSNR.

If the purpose is to show whether vp8 is superior as a *technology* to h264 CBP, then I think the comparison should use the best settings you have (ideally with a special full-on non-real time implementation) and test against the JM reference encoder. Ideally you would use the same or similar GOP structures, number of references, prediction and QP hierarchies.

Comparing different real-time implementations of different codecs trying to do high quality coding with different GOP structures and using rate control with different strategies is just a waste of time. The first two elements in the list above are alone worth a very significant amount of bit rate.

On the other hand, a quick perusal of the actual tools would suggest that vp8 and h264 CBP are likely "comparable" and the variation between implementations of the same technology would be bigger than the variation between the technologies. If we could agree that then a lot of time could be saved.

best regards

Thomas

vs:

 ./bin/vpxenc --lag-in-frames=0 --target-bitrate=${rate} --kf-min-dist=3000 \
      --kf-max-dist=3000 --cpu-used=0 --fps=${frame_rate}/1 --static-thresh=0 \
      --token-parts=1 --drop-frame=0 --end-usage=cbr --min-q=2 --max-q=56 \
      --undershoot-pct=100 --overshoot-pct=15 --buf-sz=1000 \
      --buf-initial-sz=800 --buf-optimal-sz=1000 --max-intra-rate=1200 \
      --resize-allowed=0 --drop-frame=0 --passes=1 --good --noise-sensitivity=0 \
      -w ${width} -h ${height} ${filename} --codec=vp8 \
      -o ./encoded_clips/vp8/${clip_stem}_${rate}kbps.webm \
      &>./logs/vp8/${clip_stem}_${rate}kbps.txt

Best regards
Sergio

El 03/04/2013 18:20, Adrian Grange escribió:
We have uploaded a new set of test results comparing VP8 to H.264. This latest set contains fixes for some of the problems in the previous set. We would like to extend our thanks to those who made suggestions as to how we could improve our methodology and encourage suggestions as to how we can make further improvements.

In these tests we run x264 with the "veryslow" preset and VP8 with the "good, speed 0" setting in an attempt to produce comparable results.

- A Quality comparison (psnr): http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_quality.html

- An Encode Speed comparison: http://downloads.webmproject.org/ietf_tests/vp8_vs_h264_speed.html

- A comparison of the aggregate time required to decode all of the clips in the test: http://downloads.webmproject.org/ietf_tests/vp8vsh264-decodetime.txt

The file README.txt, contained within, presents details of how to build and run the tests.

The compressed video files--the output from the quality tests--can also be downloaded:

VP8:
http://downloads.webmproject.org/ietf_tests/vp8_videos<http://downloads.webmproject.org/ietf_tests/vp8_videos/>/index.html

H.264:

Adrian Grange

rtcweb mailing list

[email-masked]<mailto:[email-masked]>

rtcweb mailing list

[email-masked]<mailto:[email-masked]>

_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 4 Apr 2013 23:48:27 +0000

> Was there any specific parameter or parameter set you were wondering about?

--nal-hrd cbr and --bitrate, who suggested those? They were not part of the suggested<http://mailman.videolan.org/pipermail/x264-devel/2013-March/009920.html> settings from x264-devel.

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Harald Alvestrand
Sent: Thursday, April 04, 2013 9:31 AM
To: [email-masked]
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded

vs:

Both have the same target bitrate and the same resolution, and neither generates periodic keyframes.

Apart from that, I think they are comparable by virtue of being the parameters that were recommended for this test by people who like this particular codec implementation.

Best regards
Sergio

VP8:

H.264:

Adrian Grange

rtcweb mailing list

rtcweb mailing list

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 05 Apr 2013 02:13:40 +0200

El 05/04/2013 1:28, Mo Zanaty (mzanaty) escribió:
>
> Realtime/low latency and constrained bitrate are obviously important 
> for the actual implementation used. Thomas was pointing out that these 
> factors have nothing to do with the codec technology itself, since 
> they are purely encoder implementation optimizations. There is nothing 
> in the VP8 or H.264 standard that uniquely provides realtime/low 
> latency or constrained bitrate. Those are attributes of encoder 
> implementations which are not part of the standard.
>
> So the question was whether we care about evaluating codec technology 
> or specific implementations. If the former, then tests should be 
> staged in the same way codec experts evaluate codec technology/tools. 
> If the latter, then tests should be staged using the target 
> implementations.
>
> I'm not aware of conferencing applications which use x264, because it 
> was designed and optimized for transcoding (dvd rips to blu-ray) not 
> conferencing. Most importantly, x264 cbr mode is inappropriate for 
> conferencing since it is for broadcast MPEG transport streams that 
> must be absolutely CBR to avoid M2TS-mux overflow or underflow, and it 
> will actually insert filler data instead of real frame data to hit the 
> CBR rate exactly. Looking at the results which show the worst H.264 
> bitrate (62% above VP8) in gipsrecstat_1280_720_50_1485kbps.mkv, there 
> is almost as much filler data as real frame data, meaning the true 
> bitrate of real frame data is almost half what is reported in the 
> results. (See attached if it makes it through.)
>

I have been using x264 with very good results in my mcu implementation, 
the parameters I use are:

     params.rc.i_rc_method           = X264_RC_ABR;
     params.rc.i_bitrate                   = bitrate;
     params.rc.i_vbv_max_bitrate = bitrate;
     params.rc.i_vbv_buffer_size   = bitrate/fps;
     params.rc.f_vbv_buffer_init    = 0;
     params.rc.f_rate_tolerance    = 0.1;
     params.b_intra_refresh           = 1;

I have a smooth and constant bitrate near the target bitrate, which is 
something I still have not been able to do with VP8 encoder yet (if 
someone nows the trick, it will be very welcome!).

Best regards
Sergio

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 05 Apr 2013 09:18:46 +0200

> implementations which are not part of the standard.

I'd challenge that assumption.
If one technology behaves much better under constrained bitrate and/or 
low latency than the other, that might just possibly be linked to the 
technology.

As an obvious example, consider B-frames; due to their basic "predict 
from the future" nature, they will incur a latency penalty if used.
A test that depends on B-frames will therefore be an invalid test for 
real-time operation.

What we can always be sure of is that when a certain quality is 
demonstrated under certain conditions, that quality is achievable under 
those conditions. We should be careful about drawing larger conclusions 
than that.

> staged in the same way codec experts evaluate codec technology/tools.

by "the same way codec experts evaluate codec technology/tools", do you 
mean the way MPEG does it?

Having just started working in MPEG, with their procedures for quality 
evaluations .... I can't say I'm terribly impressed by the evaluation 
methods used. I'm also very unimpressed with the openness of the process.

> results. (See attached if it makes it through.)
It made it through, but I can't interpret it much. Certainly sounds like 
CBR is a setting to avoid.

> While the results are bad, the methodology, effort and transparency 
> are very good (if we want to compare implementations not standards). I 
> can rerun without the bogus fillers and post the results next week, 
> unless someone else can do it faster. But as Thomas pointed out, the 
> technologies themselves are comparable as far as coding tools, so any 
> results which show significant differences are either suspect or 
> explained by differences in encoder implementations or settings not 
> the codec technology itself.
>

I've pushed (as my first contribution to the actual code; the tests 
themselves were done by other Googlers) some changes that will make it a 
little easier for new people who execute them to get predictable results 
from the tests we published.

Looking forward to more contributions!

> Mo
>
> *From:[email-masked] [mailto:[email-masked]] *On 
> Behalf Of *Leon Geyser
> *Sent:* Thursday, April 04, 2013 12:56 PM
> *To:* Thomas Davies (thdavies)
> *Cc:* [email-masked]
> *Subject:* Re: [rtcweb] New VP8 vs H.264 tests uploaded
>
>     If the purpose is to show whether vp8 is superior as a
>     *technology* to h264 CBP, then I think the comparison should use
>     the best settings you have (ideally with a special full-on
>     non-real time implementation) and test against the JM reference
>     encoder. Ideally you would use the same or similar GOP structures,
>     number of references, prediction and QP hierarchies.
>
> A graph that shows the bitrate over time for each clip could be 
> usefull to make sure that no encoder spikes the bitrate too high at 
> certain moments.
> I welcome changes to the encoder settings as long as they stay 
> realtime/low latency and constrained bitrate.
>
>     suggestions as to how we could improve our methodology and
>     encourage suggestions as to how we can make further improvements.
>
>     [email-masked]  <mailto:[email-masked]>
>
> [email-masked]  <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>

---

From: "Roy, Radhika R CIV USARMY (US)" <[email-masked]>
Date: Fri, 5 Apr 2013 14:20:09 +0000

Inline [RRR]

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Mo Zanaty (mzanaty)
Sent: Thursday, April 04, 2013 7:29 PM
To: Leon Geyser; Thomas Davies (thdavies); Adrian Grange; Cullen Jennings
(fluffy); Harald Alvestrand ([email-masked])
Cc: [email-masked]
Subject: Re: [rtcweb] New VP8 vs H.264 tests uploaded

Realtime/low latency and constrained bitrate are obviously important for the
actual implementation used. Thomas was pointing out that these factors have
nothing to do with the codec technology itself, since they are purely
encoder implementation optimizations. There is nothing in the VP8 or H.264
standard that uniquely provides realtime/low latency or constrained bitrate.
Those are attributes of encoder implementations which are not part of the
standard.

[RRR] Yes, it is dependent on the codec technology how the encoding and
decoding technology conceals the BER and hence affect the video quality. The
MOS subjective or objective coding performances video quality is also
dependent on the end-to-end latency for two-way or multipoint conversation
vs. one-way video streaming.

[RRR] The key is to capture all the parameters of video coding experimental
test environments under certain circumstances starting from back-to-back
codec to across the wide area network including wireline and wrieless. Let
us speak on these specific test environments including applications and the
corresponding video quality (subjective and/or objective). 

[RRR] Before generalization of the video performances from the specific test
results, let us understand what test results we have got for that specific
test environments.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Fri, 5 Apr 2013 16:24:29 +0000

Check your output x264 streams for fillers. If you see fillers (NAL unit type 12), you are wasting bandwidth. In the example below, almost half the bitrate was wasted on fillers and other stuff (SEI) that are not real frames. Over RTP, you would waste even more than half the link bandwidth since the small SEIs would incur the full overhead of separate RTP packets. (40 bytes of RTP/UDP/IP overhead for 10 bytes of SEI.)

Mo

From: [email-masked] [mailto:[email-masked]] On Behalf Of Sergio Garcia Murillo

I have been using x264 with very good results in my mcu implementation, the parameters I use are:

I have a smooth and constant bitrate near the target bitrate, which is something I still have not been able to do with VP8 encoder yet (if someone nows the trick, it will be very welcome!).

Best regards
Sergio

---

From: Randell Jesup <[email-masked]>
Date: Fri, 05 Apr 2013 13:43:05 -0400

On 4/5/2013 12:24 PM, Mo Zanaty (mzanaty) wrote:
>
> Check your output x264 streams for fillers. If you see fillers (NAL 
> unit type 12), you are wasting bandwidth. In the example below, almost 
> half the bitrate was wasted on fillers and other stuff (SEI) that are 
> not real frames. Over RTP, you would waste even more than half the 
> link bandwidth since the small SEIs would incur the full overhead of 
> separate RTP packets. (40 bytes of RTP/UDP/IP overhead for 10 bytes of 
> SEI.)
>

Plus SRTP authentication tag (default is 10 bytes)

These are better tests than the ones presented at IETF86, though they 
aren't perfect either of course.

So I'll note that while there may be differences (in favor of either) or 
specific implementation weaknesses (solvable), I don't think there's a 
case for saying either H.264 or VP8 are technically insufficient to be 
an MTI, and either would do to avoid negotiation failure technically.  I 
do consider h.261 to be insufficient, as both the quality level/bitrate 
and the constrained resolutions would make it unacceptable to use even 
if it avoided negotiation failure. These opinions were echoed by 
multiple people in the room (and at previous IETFs).

I think the questions on MTI all come down to non-technical issues.

-- 
Randell Jesup
[email-masked]

---

From: Leon Geyser <[email-masked]>
Date: Fri, 5 Apr 2013 20:27:10 +0200

H.261 isn't *that* bad. It sure beats Motion JPEG snd ASCII art :)

On 5 April 2013 19:43, Randell Jesup <[email-masked]> wrote:

>> Check your output x264 streams for fillers. If you see fillers (NAL unit
>> type 12), you are wasting bandwidth. In the example below, almost half the
>> bitrate was wasted on fillers and other stuff (SEI) that are not real
>> frames. Over RTP, you would waste even more than half the link bandwidth
>> since the small SEIs would incur the full overhead of separate RTP packets.
>> (40 bytes of RTP/UDP/IP overhead for 10 bytes of SEI.)
>>
>>
> specific implementation weaknesses (solvable), I don't think there's a case
> for saying either H.264 or VP8 are technically insufficient to be an MTI,
> and either would do to avoid negotiation failure technically.  I do
> consider h.261 to be insufficient, as both the quality level/bitrate and
> the constrained resolutions would make it unacceptable to use even if it
> avoided negotiation failure. These opinions were echoed by multiple people
> in the room (and at previous IETFs).
>
> ______________________________**_________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>