Subject: [rtcweb] Comments on H.264 and VP8 performance comparisons

From: Bo Burman <[email-masked]>
Date: Mon, 14 Oct 2013 21:12:08 +0000

Hi all,

We would like to counter Google's suggestion that our test has only "demonstrated that it is possible to reduce VP8's performance" (updated draft on VP8 http://datatracker.ietf.org/doc/draft-alvestrand-rtcweb-vp8/). 

In fact, what we did in our test was mostly undoing some very peculiar x264 settings made by Google in their test from April 3. By instead using the x264 settings Google themselves proposed in their earlier test (from March 12), and removing threading, the difference went down from 41% to 16%. (This is without touching the VP8 parameters.)

The last change we made was to remove the rate control from the comparison, something that is standard practice in the world of video standardization. This involved changing both the x264 and VP8 parameters. After that, the difference went down to -1%.

In summary, the following steps were taken in our comparison:

1) Downloading the latest software: 41% became 36%
2) Removing threading: 26%
3) Removing bit padding: 18%
4) Removing other differences between Google's March 12 and April 3rd tests: 16%
5) Removing rate controller: -1%

Contrary to Google's note on our test the purpose was not to "reduce the VP8 performance" but rather to present a technically correct codec comparison. Below follows a detailed description of what we found:

---

When we first saw Google's test which was posted on April 3rd, we were surprised to find that their results differed so much from our own. Whereas we got a negligible difference between VP8 and H.264 constrained baseline, Google reported that H.264 constrained baseline needed 41% more bits than VP8 for the same quality. This made us look deeper into Google's test to see how this could be explained. 

The first thing we did was to download the latest versions of all software packages. By using the newest version of x264, the difference went down from 41% to 36%.

The second thing that caught our attention was that Google's x264 test used auto threading. By omitting the parameter "--threads 1", the x264 encoder defaults to "--threads auto", which means that a large number of threads will be used to compress the image (see for instance http://mewiki.project357.com/wiki/X264_Settings). This will have the effect of decreasing the compression time at the cost of quality. The VP8 codec, on the other hand, defaults to a single thread with no quality degradation. Even when we changed the x264 configuration to use the proper threading value ("--threads 1"), the x264 codec was twice as fast as VP8, and the difference in bit rate went down from 36% to 26%. 

At this time we proceeded to only test the first 10 seconds of each sequence in order to get reasonable running times. This actually increased the difference again to 29%.

Our attention now turned to the "--nal-hrd cbr" parameter in Google's x264 April 3rd parameter list. As described at http://mewiki.project357.com/wiki/X264_Settings#nal-hrd, this setting will pack the bitstream with padding bits in order to exactly reach a particular bit rate. This is useful in some circumstances such as in Blu-ray or ISDN video telephony which has to be exactly, say, 64000 bits per second, but it is undesirable in a codec comparison since it will only add bits and not increase quality. The VP8 encoder does not do such bit padding and was thus at an advantage. Removing the "--nal-hrd cbr" parameter for x264 avoided the unnecessary bit padding and the difference now went down from 29% to 18%.

At this point in time we tried removing the remaining variables that differed between the x264 parameter set of Google's first test (from March 12) and the x264 parameter set of Google's second test (from April 3rd), and this resulted in a further decrease from 18% to 16%.

Finally we removed the rate controller from the test and instead used fixed QP. As we have argued previously, having a rate controller in the loop just adds noise to the test and also risks measuring the performance of the rate controller rather than the codec. Using fixed QP (no rate control) is therefore established practice in the video codec community. As an example, the Motion Pictures Expert Group (MPEG) recommends using fixed QP for video comparisons, see for instance (http://mpeg.chiariglione.org/standards/exploration/internet-video-coding/call-proposals-internet-video-coding-technology). The reworked test without rate control (which we published on June 22) then got the result of -1%. This means H.264 constrained baseline (in the x264 implementation) outperformed VP8. We also found H.264 constrained high, again using the x264 implementation, to be 24% better than VP8.

Google's comparison on April 3rd also included a speed test. In this, x264 is set to use only one thread, the parameter "--threads 1" is set. This means that x264 cannot enjoy the faster speed of a parallel implementation. We do not quite understand this choice of parameters from Google: When threading should be avoided (in the case of measuring quality), it is used, whereas when it would be helpful (in the case of measuring speed), it is avoided. In both cases this is unfavorable to x264. This does not seem entirely fair.

Note that the biggest differences in performance were not due the changes made to the VP8 settings but to those of x264: The threading and the bit padding alone accounted for 21 of the 41 percentage points. Thus, we do not think that our test is about trying to "reduce the VP8 performance". Instead, the major things have been avoiding what we believe are unfortunate parameter choices for x264 on behalf of Google, and to create a technically correct test.

Best Regards,

Bo Burman

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 15 Oct 2013 15:37:33 +0200

Thanks for the details, Bo!

We had the discussion about locking QP at MPEG too; the H.264 reference 
model is usually run with a fixed QP per frame type (the QP for the I 
frame is different from the P frame, which is again different from the B 
frame if used, but in encoding to a specific bitrate, they change the QP 
once per run so that the data rate comes out right), which is fine for 
comparing two minor versions of a codec to , but doesn't seem equally 
obvious when comparing two codecs that have differing histories and are 
expected to behave differently on a large number of points.
The VP8 codec was designed and tested in a real-world environment where 
rate controls have to be used, and where we can't make repeated passes 
over the clips to find the best QP values to use for a given clip. This 
colors the way it behaves; there's no reason to believe it would be 
optimal in a fixed-QP environment.

When we debated this at MPEG, the MPEG conclusion was that they accepted 
that we would encode the VP8 clips with our normal rate control despite 
the strenous objections of some members - I don't want to second-guess 
MPEG's decisions in this forum.

I'll look at the changes you're proposing. We did do one set of tweaks 
"as suggested online" to the H.264 parameters, the diff (from the public 
repo of our testing code) is:

Date:   Tue Mar 19 10:22:28 2013 -0700

     Used settings suggested by x264 devs and updated vp8 settings too

     We didn't go down to the vp8 equivalent of very_slow for quality tests.
     Now we do also adjusted the x264 settings to match what was 
suggested on
     line.

     Change-Id: I2270d156cbe5893ddb7d37633f5fe61f93fae091

diff --git a/run_h264_tests.sh b/run_h264_tests.sh
index a8a1db0..6d75136 100755
--- a/run_h264_tests.sh
+++ b/run_h264_tests.sh
@@ -50,9 +50,11 @@ do
    for (( rate=rate_start; rate<=rate_end; rate+=rate_step ))
    do
      # Encode into 
./<clip_name>_<width>_<height>_<frame_rate>_<rate>kbps.yuv
-    x264 --vbv-bufsize ${rate} --bitrate ${rate} --fps ${frame_rate} \
+    x264 --nal-hrd cbr --vbv-maxrate ${rate} --vbv-bufsize ${rate} \
+      --vbv-init 0.8 --bitrate ${rate} --fps ${frame_rate} \
        --profile baseline --no-scenecut --keyint infinite --preset 
veryslow \
        --input-res ${width}x${height} \
+      --tune psnr \
        -o ./encoded_clips/h264/${clip_stem}_${rate}kbps.mkv ${filename} \
        2> ./logs/h264/${clip_stem}_${rate}kbps.txt

We don't seem to have made any changes since then; I'll experiment with 
the settings you're suggesting.

FWIW, when I run the head of our repository (which is locked to a 
specific software rev for each codec; I'll play with those too), the 
summary report says that the difference is 71%. Feel free to check out 
the latest version!

On 10/14/2013 11:12 PM, Bo Burman wrote:
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 22 Oct 2013 11:49:45 +0200

> 5) Removing rate controller: -1%
Just a quick update on this - I did not manage to get a new draft ready 
before the deadline, so I'll have to resort to sending email:

I applied steps 1), 2) and 3) to the repository mentioned in the draft. 
The numbers I got were different, but significant. Below are the 
VP8-to-x264 differences I encountered each step of the way.

- Master branch before October 15: Difference 71.52%
- Updating x264 from 198a7ea (aug 16 2012) to c832fe (March 1 2013): 
Difference 71.44%
   (I could not go beyond this because yasm 1.2 was required for newer 
versions).
- Adding the --thread 1 parameter: Difference 26.11%
- Removing the --nai-hrd=cbr parameter that was suggested by x264 
people: Difference 13.87%
- Removing vbv-maxrate and vbv-init 0.8: Difference 12.74%

I did not shorten the clips to 10 seconds, nor did I try to control rate 
by constant cq instead of a rate controller; if Bo's numbers are right, 
this shouldn't matter.

I did try removing "vbv-bufsize ${rate}", since this was the remaining 
difference I could find for Bo's point 4 "other differences", but that 
was actually harmful (difference increased to 19.05%), so I put it back.

I checked this in to the repository - the commit is here:

http://git.chromium.org/gitweb/?p=webm/vpx_codec_comparison.git;a=commitdiff;h=ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5

This number does not fit the impression the video codec team has from 
other tests - they think
VP8 can do a lot more than 13% better than baseline - but at the draft 
deadline, we had not found the
set of VP8 parameters that showed this for this particular clip set.

Commentary: I'm surprised at x264's choice of default value for the 
--thread parameter. Accepting a 26% bitrate hit seems like a large price 
to pay for going faster by default. Is there a bug here?

---

From: Bo Burman <[email-masked]>
Date: Tue, 22 Oct 2013 14:20:39 +0000

Hi all,

We understand that there are many people that are interested in a VP8 vs x264 comparison that includes rate controls.

We therefore performed a quick study of the VP8 and x264 rate controls to see whether there were any major differences between them in the way they select quantizers. We found that VP8 uses what can be described as QP-toggling: By lowering the QP of, say, every 6th frame, you increase the quality of these markedly. The following frames also benefit from the increased quality (even though they have to make do with less bits than before) since they can predict from the high-quality frame. 

However, we found that the rate-control of x264 does not QP-toggle in this fine granular way, and x264 may therefore be at a disadvantage when comparing against VP8. So we ran a test using a modified version of x264 to allow QP-toggling during rate control. The modification is very simple and a patch for the x264 code is found in the bottom of this email.

With this change, H.264 baseline (implemented using the modified x264 with rate control turned on) performs equally to VP8 with rate-control on; the difference is 1.3% in favor of VP8, but if you swap the order of the two codecs in draw_graphs.sh, H.264 baseline wins with 1.6%. This result is thus a tie between VP8 and H.264 baseline, and this is consistent with our previous fixed-QP test, and far from the (at least) 13% advantage for VP8 stated by Google.

The modification is simple and it may well be possible to improve x264 by more sophisticated modifications. But pursuing this track is what we oppose, we do not want comparisons where the outcome depends on how well you modify a particular rate control mechanism - the rate control is  codec-agnostic after all. To avoid this we still maintain our view that codec comparisons should be done using fixed QP settings, which is also the practice in MPEG and VCEG.

We used the test parameters from Google's test repository  https://git.chromium.org/git/webm/vpx_codec_comparison.git, the version ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5.

Here are the x264 and comparison script patches:
--

diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c
index 1a39a49..2cf256f 100644
--- a/encoder/ratecontrol.c
+++ b/encoder/ratecontrol.c
@@ -1707,7 +1707,18 @@ int x264_ratecontrol_mb_qp( x264_t *h )
{
     x264_emms();
     float qp = h->rc->qpm;
-    if( h->param.rc.i_aq_mode )
+
+       if (h->sh.i_type != SLICE_TYPE_I){
+          if ((h->fenc->i_poc % 12 )== 0)
+              qp -= 3.0;
+          else
+              qp += 0.0;
+   
+       } else {
+          qp -= 5.0;
+       }
+
+    if( h->param.rc.i_aq_mode)
     {
          /* MB-tree currently doesn't adjust quantizers in unreferenced frames. */
         float qp_offset = h->fdec->b_kept_as_ref ? h->fenc->f_qp_offset[h->mb.i_mb_xy] : h->fenc->f_qp_offset_aq[h->mb.i_mb_xy];

draw_graphs.sh:

change 
./visual_metrics.py metrics_template.html "*0.txt" stats/h264 stats/vp8  > vp8_vs_h264_quality.html
to
./visual_metrics.py metrics_template.html "*0.txt" stats/vp8 stats/h264  > vp8_vs_h264_quality2.html

Best Regards,
Bo Burman

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Harald Alvestrand
> Sent: den 22 oktober 2013 11:50
> To: [email-masked]
> Subject: Re: [rtcweb] Comments on H.264 and VP8 performance comparisons
> 
> > We would like to counter Google's suggestion that our test has only "demonstrated that it is possible to reduce VP8's
> performance" (updated draft on VP8 http://datatracker.ietf.org/doc/draft-alvestrand-rtcweb-vp8/).
> >
> > In fact, what we did in our test was mostly undoing some very peculiar
> > x264 settings made by Google in their test from April 3. By instead
> > using the x264 settings Google themselves proposed in their earlier
> > test (from March 12), and removing threading, the difference went down
> > from 41% to 16%. (This is without touching the VP8 parameters.)
> >
> > The last change we made was to remove the rate control from the comparison, something that is standard practice in
> the world of video standardization. This involved changing both the x264 and VP8 parameters. After that, the difference
> went down to -1%.
> >
> Just a quick update on this - I did not manage to get a new draft ready before the deadline, so I'll have to resort to
> sending email:
> 
>    (I could not go beyond this because yasm 1.2 was required for newer versions).
> I did not shorten the clips to 10 seconds, nor did I try to control rate by constant cq instead of a rate controller; if Bo's
> numbers are right, this shouldn't matter.
> 
> I did try removing "vbv-bufsize ${rate}", since this was the remaining difference I could find for Bo's point 4 "other
> differences", but that was actually harmful (difference increased to 19.05%), so I put it back.
> 
> This number does not fit the impression the video codec team has from other tests - they think
> VP8 can do a lot more than 13% better than baseline - but at the draft deadline, we had not found the set of VP8
> parameters that showed this for this particular clip set.
> 
> Commentary: I'm surprised at x264's choice of default value for the --thread parameter. Accepting a 26% bitrate hit
> seems like a large price to pay for going faster by default. Is there a bug here?
>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 22 Oct 2013 16:48:57 +0200

> We used the test parameters from Google's test repository  https://git.chromium.org/git/webm/vpx_codec_comparison.git, the version ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5.

Thanks!

Can you verify that you did see the same 12.7% difference when you ran 
the unmodified ada7d checkout? I want to be sure that I haven't missed 
any external influences on the results.

Having a 2.9% difference by swapping the order of arguments seems wrong; 
I'll pass this on to the guy who wrote that particular script.

---

From: Bo Burman <[email-masked]>
Date: Tue, 22 Oct 2013 16:07:06 +0000

I can confirm we get 12.7% advantage for VP8 with the unmodified ada7d checkout.

> From: Harald Alvestrand [mailto:[email-masked]]
> Sent: den 22 oktober 2013 16:49
> To: Bo Burman; [email-masked]
> > We therefore performed a quick study of the VP8 and x264 rate controls to see whether there were any major
> differences between them in the way they select quantizers. We found that VP8 uses what can be described as QP-
> toggling: By lowering the QP of, say, every 6th frame, you increase the quality of these markedly. The following frames
> also benefit from the increased quality (even though they have to make do with less bits than before) since they can
> predict from the high-quality frame.
> >
> > However, we found that the rate-control of x264 does not QP-toggle in this fine granular way, and x264 may therefore
> be at a disadvantage when comparing against VP8. So we ran a test using a modified version of x264 to allow QP-toggling
> during rate control. The modification is very simple and a patch for the x264 code is found in the bottom of this email.
> >
> > With this change, H.264 baseline (implemented using the modified x264 with rate control turned on) performs equally
> to VP8 with rate-control on; the difference is 1.3% in favor of VP8, but if you swap the order of the two codecs in
> draw_graphs.sh, H.264 baseline wins with 1.6%. This result is thus a tie between VP8 and H.264 baseline, and this is
> consistent with our previous fixed-QP test, and far from the (at least) 13% advantage for VP8 stated by Google.
> >
> > The modification is simple and it may well be possible to improve x264 by more sophisticated modifications. But
> pursuing this track is what we oppose, we do not want comparisons where the outcome depends on how well you
> modify a particular rate control mechanism - the rate control is  codec-agnostic after all. To avoid this we still maintain our
> view that codec comparisons should be done using fixed QP settings, which is also the practice in MPEG and VCEG.
> >
> > We used the test parameters from Google's test repository
> https://git.chromium.org/git/webm/vpx_codec_comparison.git, the version
> ada7d7937a54e47fc18e2e0a1287aea29dc5d1c5.
> 
> Thanks!
> 
> Can you verify that you did see the same 12.7% difference when you ran the unmodified ada7d checkout? I want to be
> sure that I haven't missed any external influences on the results.
> 
> Having a 2.9% difference by swapping the order of arguments seems wrong; I'll pass this on to the guy who wrote that
> particular script.
> 
> >
> > diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c index
> > 1a39a49..2cf256f 100644
> > @@ -1707,7 +1707,18 @@ int x264_ratecontrol_mb_qp( x264_t *h ) {
> >       x264_emms();
> >       float qp = h->rc->qpm;
> >           float qp_offset = h->fdec->b_kept_as_ref ?
> > h->fenc->f_qp_offset[h->mb.i_mb_xy] :
> > h->fenc->f_qp_offset_aq[h->mb.i_mb_xy];
> >
> >
> > draw_graphs.sh:
> >
> > change
> > ./visual_metrics.py metrics_template.html "*0.txt" stats/h264
> > stats/vp8  > vp8_vs_h264_quality.html to ./visual_metrics.py
> > metrics_template.html "*0.txt" stats/vp8 stats/h264  >
> > vp8_vs_h264_quality2.html
> >
> > Best Regards,
> > Bo Burman
> >
> >>> We would like to counter Google's suggestion that our test has only
> >>> "demonstrated that it is possible to reduce VP8's
> >>> test (from March 12), and removing threading, the difference went
> >>> down from 41% to 16%. (This is without touching the VP8 parameters.)
> >>>
> >>> The last change we made was to remove the rate control from the
> >>> comparison, something that is standard practice in
> >> the world of video standardization. This involved changing both the
> >> x264 and VP8 parameters. After that, the difference went down to -1%.
> >>> 4) Removing other differences between Google's March 12 and April
> >>> 3rd
> >>> tests: 16%
> >> Just a quick update on this - I did not manage to get a new draft
> >> ready before the deadline, so I'll have to resort to sending email:
> >>
> >> I did not shorten the clips to 10 seconds, nor did I try to control
> >> rate by constant cq instead of a rate controller; if Bo's numbers are right, this shouldn't matter.
> >>
> >> I did try removing "vbv-bufsize ${rate}", since this was the
> >> remaining difference I could find for Bo's point 4 "other differences", but that was actually harmful (difference
> increased to 19.05%), so I put it back.
> >>
> >> http://git.chromium.org/gitweb/?p=webm/vpx_codec_comparison.git;a=com
> >> mitdiff;h=ada7d7937a54e47fc18e2e0a1287
> >> aea29dc5d1c5
> >>
> >>
> >> VP8 can do a lot more than 13% better than baseline - but at the
> >> draft deadline, we had not found the set of VP8 parameters that showed this for this particular clip set.
> >>
> >> --thread parameter. Accepting a 26% bitrate hit seems like a large price to pay for going faster by default. Is there a
> bug here?
> >>

---

From: Bo Burman <[email-masked]>
Date: Tue, 22 Oct 2013 16:26:37 +0000

Hi all, 

On 10/22/2013 04:28 PM, Harald Alvestrand wrote:

> Having a 2.9% difference by swapping the order of arguments seems wrong; I'll pass this on to the guy who wrote that particular script.

Actually, this is not so weird at all. With the way of measuring that is done in your script, it makes a huge difference which sequence is the "anchor" (the first argument) and which is the "test subject" (the second argument). The reason is that a 10% bit rate increase is not equivalent to a 10% decrease when you turn the arguments around. An illustrative example:

Assume there are only two sequences, one that VP8 is really good at, so H264 need twice the bit rate to reach the same quality, and one that H.264 is really good at, so that it only need half the bits to reach the same quality. Then, using the VP8 codec as the anchor you get:

Sequence 1: H.264 needs twice the number of bits (+100%) 
Sequence 2: H.264 needs half the number of bits (-50%) 
You average these two and you get (100%-50%)/2 = 25% more bits on average for H.264

If you, on the other hand, use H.264 as the anchor (first argument), you get

Sequence 1: VP8 needs half the number of bits (-50%) 
Sequence 2: VP8 needs twice the number of bits (+100%) 
You average these two and you get (-50% + 100%)/2 = 25% more bits on average for VP8

Hence the script is biased towards the first argument in the script.

Best Regards
Bo Burman

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 23 Oct 2013 11:26:03 +0200

On 10/22/2013 06:26 PM, Bo Burman wrote:
> Actually, this is not so weird at all. With the way of measuring that is done in your script, it makes a huge difference which sequence is the "anchor" (the first argument) and which is the "test subject" (the second argument). The reason is that a 10% bit rate increase is not equivalent to a 10% decrease when you turn the arguments around.

Thanks for that explanation - makes sense!
Also makes sense that it wouldn't matter for small differences, or ones 
where the differences were roughly the same for all the clips - the case 
we're in, where clips vary widely in relative performance, is a bit special.

BTW, I talked to one of the guys who did the June tweaks to x264 
parameters yesterday - he explained that he had done a (relatively) 
constant bitrate encoding for VP8 (mode=cbr), and was trying to achieve 
roughly the same result for x264; the advice he was given was to use the 
parameters you saw, with vbv-maxrate ${rate} and vbv-init 0.8 . Both VP8 
and x264 seem to gain about a percentage point if they are allowed to do 
a more variable bitrate, so this is not a huge differentiator.

>   An illustrative example: