Subject: [rtcweb] VP8 vs H.264 - subjective evaluation

From: Harald Alvestrand <[email-masked]>
Date: Tue, 22 Oct 2013 12:24:52 +0200

In my VP8 advocate draft, I referred to a subjective evaluation test
(test with
actual human viewers) done between VP8 and H.264 Baseline by a neutral
(non-Google) laboratory.

The attached presentation is the writeup of the results of that test.

-- 
Surveillance is pervasive. Go Dark.

---

From: Krasimir Kolarov <[email-masked]>
Date: Tue, 22 Oct 2013 16:46:29 -0700

To be very clear for the IETF community:

As Harald’s slide deck notes on Slide 2, these subjective test results were not presented nor discussed at MPEG meetings. Hence they only reflect the opinion of the VP8 proponents.

The only subjective tests that were presented at the meeting compared VP8 and IVC (the Internet Video Coding proposal considered by Mpeg, which is different than AVC Constrained Baseline). 
Those tests raised a lot of controversy during discussions at the meeting and in fact some of the results had to be excluded because of significant differences in comparison conditions (rate control, etc.). This is not unlike recent discussions on this reflector about comparisons of VP8 and H.264 Constrained Baseline. 

Krasimir

On Oct 22, 2013, at 3:24 AM, Harald Alvestrand <[email-masked]> wrote:

> <VP8 vs AVC Baselinev2.pdf>_______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Basil Mohamed Gohar <[email-masked]>
Date: Tue, 22 Oct 2013 20:40:23 -0400

> The attached presentation is the writeup of the results of that test.

Are the sources used in the video comparison test available anywhere?

-- 
Libre Video
http://librevideo.org

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 23 Oct 2013 08:04:21 +0200

> As Harald’s slide deck notes on Slide 2, these subjective test results were not presented nor discussed at MPEG meetings. Hence they only reflect the opinion of the VP8 proponents.

Well.... strictly speaking, the results reflect the work of Vittorio 
Baronici.
The methods reflect the methods chosen by MPEG.
The presentation was made by a VP8 proponent.

> Those tests raised a lot of controversy during discussions at the meeting and in fact some of the results had to be excluded because of significant differences in comparison conditions (rate control, etc.).

Krasimir, I think you're mixing up two debates here (easy, since there 
have been two sets of subjective evaluation tests).

The use of rate control was accepted by MPEG, despite protests by some 
of the members, since this is part of the design of VP8 in a way that it 
was never part of the design of H.264.

No result was ever excluded in the IVC to VP8 comparision. After 
discussion, we all agreed that the results needed to be presented with a 
horizontal axis representing actual rate, not target rate; this has been 
done in the slides I sent out.

Two results were dropped from the presentation of the VP8 results, which 
were not presented as a comparision with AVC High profile, due to issues 
with achieving very low bitrates.

>   This is not unlike recent discussions on this reflector about comparisons of VP8 and H.264 Constrained Baseline.

Agreed. In both cases, the protestations that rate control is a problem 
reflect a position that MPEG did not come to consensus on. MPEG accepted 
the results that were produced using rate control.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 23 Oct 2013 10:26:19 +0200

> Are the sources used in the video comparison test available anywhere?
They are under an MPEG license, so they're freely available to MPEG 
members for use in developing MPEG standards. If you fall within that 
category, I can send you the FTP server and password.

MPEG doesn't have the tradition of open materials that the IETF has.

---

From: Basil Mohamed Gohar <[email-masked]>
Date: Wed, 23 Oct 2013 09:32:16 -0400

> MPEG doesn't have the tradition of open materials that the IETF has.

Well, that's unfortunate.  Thanks for clarifying, though.

It would be nice if, when time permits, a similar test can be made,
using the same criteria (encoding parameters, etc.), but with known
clips (e.g., from derf's collection, which has everything, including
both SD and HD clips).  Publishing the versions and exact encoding
parameters (including command line usage) would be helpful to that aim,
I suppose.

-- 
Libre Video
http://librevideo.org

---

From: Bo Burman <[email-masked]>
Date: Thu, 24 Oct 2013 08:49:25 +0000

It is really difficult to at all comment on this since there is far too little data available. There is no way to review the settings and parameters used, and of course no way to repeat the tests, and hopefully our previous inputs related to objective quality evaluation has made it clear to everyone that parameter settings make a huge difference.

It can also be noted that even if the results are assumed to be correct (which there is currently no way to verify) the conclusions drawn seem incorrect. E.g., it is stated that "In 7/10 sequences tested VP8 LD was clearly better than AVC constrained baseline", but when examining the plots the confidence intervals seem to be overlapping.

Without any more information it can only be concluded that this test has very little value.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Basil Mohamed Gohar
> Sent: den 23 oktober 2013 15:32
> To: [email-masked]
> Subject: Re: [rtcweb] VP8 vs H.264 - subjective evaluation
> 
> >>> (test with actual human viewers) done between VP8 and H.264 Baseline
> >>> by a neutral
> >>> (non-Google) laboratory.
> >>>
> It would be nice if, when time permits, a similar test can be made, using the same criteria (encoding parameters, etc.),
> but with known clips (e.g., from derf's collection, which has everything, including both SD and HD clips).  Publishing the
> versions and exact encoding parameters (including command line usage) would be helpful to that aim, I suppose.
> 
> --
> Libre Video
> http://librevideo.org
> _______________________________________________
> rtcweb mailing list
> [email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 24 Oct 2013 13:32:58 +0200

> It is really difficult to at all comment on this since there is far too little data available. There is no way to review the settings and parameters used, and of course no way to repeat the tests,…

Bo, your organization is an MPEG member, so you can get the streams, and
the scripts used to produce them, any day you want.
I've sent you the login details to the server in separate mail.

The VP8 parameters were the ones used for the VP8 vs IVC comparision
presented in MPEG input documents to the Geneva MPEG meeting; I think
you will find the actual command lines either in the input documents or
in the attachments.

The AVC streams used were the ones produced for the initial MPEG
"royalty-free video codec" call for proposals 2 years back, using the JM
encoder; I don't know who produced them, it might even have been
Ericsson. I'd expect that you'd find the encoding parameters used in
those documents.

If you want to repeat the tests, feel very free to hire your own
independent laboratory to do so - Vittorio's evaluation methods are very
well known in the MPEG community.

We have absolutely no interest in keeping anything secret here - the
only reason it's not all in the open is because of MPEG rules about
access to documents and test sequences!

> It can also be noted that even if the results are assumed to be correct (which there is currently no way to verify) the conclusions drawn seem incorrect. E.g., it is stated that "In 7/10 sequences…

I'm not sure what you are looking at - could you identify the slides
where you see overlapping confidence intervals between the red line (VP8
LD) and the green line (AVC constrained baseline)?

> Without any more information it can only be concluded that this test has very little value.

As I've noted in another thread, discussions about quality seem to have
very little value anyway, since there's nobody who's said that they will
change their mind if they get some new quality information. But we had
this evaluation already, so sharing information with the community
seemed to be the community-minded thing to do.