Subject: [rtcweb] Agenda time request for draft-dbenham-webrtcvideomti

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 26 Feb 2013 18:08:26 +0000

I would like 25 minutes for presentation of information key to draft-dbenham-webrtcvideomti that is not interrupted and 15 minutes for Q/A for the work. 

We have spent many hours of meeting time discussing the way we were going to make a decision around codecs and information needed, but we have not yet spent time discussing the actually information to make the decision. This draft addressees several very key issues, including the actually quality comparison of VP8 and H.264. I think that spending any less time than this would result in a situation where the WG was not making an decisions with the information available. Similarly, other  people that have written other drafts with similar or different views also need to speak up about how much time they need and that we make sure everyone has time to explain the key information, people have time to ask questions about it, and we can make an informed decisions instead of having to discuss the video codecs at many future meetings.

---

From: "Matthew Kaufman (SKYPE)" <[email-masked]>
Date: Tue, 26 Feb 2013 18:47:29 +0000

Where is said draft? Hard to know if this would be time well spent if the draft isn't available. http://tools.ietf.org/id/draft-dbenham-webrtcvideomti-00 returns 404

Matthew Kaufman

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Cullen Jennings (fluffy)
Sent: Tuesday, February 26, 2013 10:08 AM
To: <[email-masked]>
Subject: [rtcweb] Agenda time request for draft-dbenham-webrtcvideomti

_______________________________________________
rtcweb mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 26 Feb 2013 18:54:12 +0000

sorry - I put the wrong draft name, it got updated to a -01 

better link at 

http://bit.ly/13fOrrZ

---

From: "Matthew Kaufman (SKYPE)" <[email-masked]>
Date: Tue, 26 Feb 2013 18:57:45 +0000

Interesting... http://bit.ly/15gL7Lw

(Which at least for me gave a page of 5 results, all of them pointers to your original mailing list posting or the agenda with the wrong link in it, none to the draft)

Matthew Kaufman

-----Original Message-----
From: Cullen Jennings (fluffy) [mailto:[email-masked]] 
Sent: Tuesday, February 26, 2013 10:54 AM
To: Matthew Kaufman (SKYPE)
Cc: <[email-masked]>
Subject: Re: Agenda time request for draft-dbenham-webrtcvideomti

better link at 

http://bit.ly/13fOrrZ

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 26 Feb 2013 11:00:56 -0800

http://tools.ietf.org/html/draft-dbenham-webrtc-videomti-01

On 02/26/2013 10:57 AM, Matthew Kaufman (SKYPE) wrote:
> (Which at least for me gave a page of 5 results, all of them pointers to
> your original mailing list posting or the agenda with the wrong link in it,
> none to the draft)
> 
> Matthew Kaufman
> 
> -----Original Message----- From: Cullen Jennings (fluffy)
> [mailto:[email-masked]] Sent: Tuesday, February 26, 2013 10:54 AM To:
> Matthew Kaufman (SKYPE) Cc: <[email-masked]> Subject: Re: Agenda time
> request for draft-dbenham-webrtcvideomti
> 
> 
>> Where is said draft? Hard to know if this would be time well spent if the
>> draft isn't available.
>> http://tools.ietf.org/id/draft-dbenham-webrtcvideomti-00 returns 404
>> 
>> Matthew Kaufman
>> 
>> -----Original Message----- From: [email-masked]
>> [mailto:[email-masked]] On Behalf Of Cullen Jennings (fluffy) 
>> Sent: Tuesday, February 26, 2013 10:08 AM To: <[email-masked]> Subject:
>> [rtcweb] Agenda time request for draft-dbenham-webrtcvideomti
>> 
>> 
>> I would like 25 minutes for presentation of information key to
>> draft-dbenham-webrtcvideomti that is not interrupted and 15 minutes for
>> Q/A for the work.
>> 
>> We have spent many hours of meeting time discussing the way we were going
>> to make a decision around codecs and information needed, but we have not
>> yet spent time discussing the actually information to make the decision.
>> This draft addressees several very key issues, including the actually
>> quality comparison of VP8 and H.264. I think that spending any less time
>> than this would result in a situation where the WG was not making an
>> decisions with the information available. Similarly, other  people that
>> have written other drafts with similar or different views also need to
>> speak up about how much time they need and that we make sure everyone has
>> time to explain the key information, people have time to ask questions
>> about it, and we can make an informed decisions instead of having to
>> discuss the video codecs at many future meetings.
>> 
>> 
>> _______________________________________________ rtcweb mailing list 
>> [email-masked] https://www.ietf.org/mailman/listinfo/rtcweb
>> 
> 
> 
> _______________________________________________ rtcweb mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/rtcweb
> 

- -- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: http://blog.marc.petit-huguenin.org
Profile: http://www.linkedin.com/in/petithug

---

From: Michael Procter <[email-masked]>
Date: Tue, 26 Feb 2013 19:16:45 +0000

I imagine it is this one:

http://tools.ietf.org/id/draft-dbenham-webrtc-videomti-01.txt

Michael

---

From: Adam Roach <[email-masked]>
Date: Tue, 26 Feb 2013 14:29:08 -0600

I want to caution people about what they are -- and are not -- seeing in 
the demonstration videos pointed to by this draft. I don't think the 
draft is intentionally deceptive on this point, but a casual reader 
might fail to make a somewhat subtle (but catastrophically important) 
distinction.

The videos in section 4 are a pretty convincing comparison between 
hardware encoding and software encoding on same-class platforms. They 
happen to use different codecs, but that's a non-factor compared to the 
overwhelming difference between encoding horsepower.

What they absolutely don't show is a comparison of H.264 to VP8. I don't 
beleive the authors intend to do so, but other parties might be tempted 
to construe the videos in that light. Doing so would be a 
straightforward application of "trick 3b" from "How to cheat on video 
encoder comparisons" (http://x264dev.multimedia.cx/archives/472). Of 
*course* coding on a DSP will look better than coding on a CPU that 
draws roughly the same power.

I'm not discounting the availability of H.264 hardware encoders as a 
factor, and the videos do a pretty good job of showing off the 
advantages of hardware encoding. But that's all they show.

/a

---

From: Randell Jesup <[email-masked]>
Date: Wed, 27 Feb 2013 10:29:11 -0500

On 2/26/2013 3:29 PM, Adam Roach wrote:
> I want to caution people about what they are -- and are not -- seeing 
> in the demonstration videos pointed to by this draft. I don't think 
> the draft is intentionally deceptive on this point, but a casual 
> reader might fail to make a somewhat subtle (but catastrophically 
> important) distinction.
>
> happen to use different codecs, but that's a non-factor compared to 
> the overwhelming difference between encoding horsepower.

Not only that, but you're comparing VERY different applications with no 
synchronization of settings, of network bandwidth, etc.  Bria/VP8 vs 
Facetime/H.264-more-than-baseline-and-not-even-proposed-here? Not a 
useful comparison.  Similar arguments for all the demos (I won't call 
them tests).

I looked at these originally.  I've worked with Bria in the past. I'm 
certain the bandwidth usage was different.  I'm sure the bandwidth 
adaptation is different.  Application logic is different. Delay almost 
certainly has nothing to do with the codec, but is largely the result of 
jitter buffers/avsync and other issues in the app/browser/os.  Etc, etc.

You can't even begin a comparison of *codecs* without normalizing 
settings *and* measuring how much they actually use, loss, etc.  For 
another example (on the other side of this argument), see the videos 
posted by Google during the last IETF - at the low end of bandwidth 
settings (and this actually is a controlled test with settings), VP8 
looks FAR better.  Until I looked closer, and asked some questions: it 
turned out that while both were *set* for 100Kbps (IIRC), the VP8 
bitrate controller decided it would look too bad or that the bitrate was 
too low for that res/framerate, and ran closer to 200K.

>
> What they absolutely don't show is a comparison of H.264 to VP8. I 
> don't beleive the authors intend to do so, but other parties might be 
> tempted to construe the videos in that light. Doing so would be a 

And I'm not even sure they show the advantage of hardware encoding.

*If* we're going to discuss codec quality, we need (much) better data 
than this.  Google's comparisons from the last IETF are much more 
scientific (though with the caveat I mentioned you have to watch out 
for).  I'll also assert that if the difference is minor (+-10 or 20% 
either way - number pulled out of air) it's irrelevant for this 
discussion.  Even major differences would require discussion as to 
whether they matter.  I personally don't see a need to discuss it, 
unless someone has hard facts that one or the other falls down badly by 
comparison - and that we don't believe it's something that can be fixed 
(especially if it's an implementation flaw).

-- 
Randell Jesup
[email-masked]

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 27 Feb 2013 12:28:51 -0500

>(especially if it's an implementation flaw).

So if we're not too fussy about quality/performance, what are the criteria
for choosing a video codec?

...Paul

---

From: "Matthew Kaufman (SKYPE)" <[email-masked]>
Date: Wed, 27 Feb 2013 17:41:50 +0000

Given that some vendors are certain they will not be shipping one, and other vendors are certain they will not be shipping the other, I'm not sure why we would have the discussion at all... but certainly *quality* isn't the axis they're using to make these decisions.

I think the draft is great *except* for the quality sections... remove those and we might be able to use it as a great jumping-off point for the actual conversation.

Matthew Kaufman

---

From: Adam Roach <[email-masked]>
Date: Wed, 27 Feb 2013 11:45:32 -0600

> for choosing a video codec?

When trying to figure out what a working group is *supposed* to be 
doing, I'd generally start with its charter: " the working group will 
try to avoid encumbered technologies that require royalties or other 
encumbrances that would prevent such technologies from being easy to use 
in web browsers."

/a

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 27 Feb 2013 17:48:42 +0000

Hi Adam,

I generally agree with your comments, and also with Randell's comments
posted somewhat later in this thread.  Unfortunately, there is no good way
to compare bitstream efficiency (in contrast to encoder quality) without
theoretical analysis.  A few bits and pieces of theoretical analysis
exist, have been cited in this discussion already, and generally come out
in favor of H.264 (even baseline), but they are arguably tainted by their
respective authors' commercial interest.

That said, I somewhat disagree with your comment on the influence of
hardware support for the moderate complexity of H.264 or VP8.  Yes, the
search tree of the syntax even of a 10 year old standard like H.264 is so
large that you can max out a fairly large number of any CPU or DSP or FPGA
in existence today.  However, the quality gain you get by throwing really
large amounts of processing power to an encoder, over what is common on a
desktop PC today, is rather limited.  For video conferencing content, I
would wager a guess that it is less than 1 dB.  That is an invisible
quality difference to almost all non-expert viewers.  The reason is that
those encoders nowadays got their search and mode decision heuristics
right.  Mode decision and motion search rarely take more than a few dozen
per cent of the overall cycle budget.  The rest goes to things like DTC
and entropy coding, and is dependent only on factors chosen by the
application (picture size, frame rate, bitrate).  So for the purpose of
comparing real-time H.264-generation video encoding with picture sizes up
to, say, 720p60, it does hardly matter whether you run your encoder on a
dedicated platform or an a desktop class CPU.  And we are very close to
the time where it does not matter any more for 1080p60, either.

For modern codecs, like H.265, some hardware acceleration will continue to
be desirable for a few years, especially with picture sizes of 4K and
beyond.  But eventually, and I believe almost certainly over the
commercial lifetime of the standard, general purpose CPUs will catch up.

Finally, no one I know uses DSPs for video encoding anymore.  Those
dedicated hardware H.264 encoders are almost all hardware-acceleration
units for things like motion search and CABAC, placed around a fairly
powerful general purpose CPU that runs heuristics not much different from
what a pure software implementation uses.

Stephan 

On 2.26.2013 12:29 , "Adam Roach" <[email-masked]> wrote:

>> We have spent many hours of meeting time discussing the way we were
>>going to make a decision around codecs and information needed, but we
>>have not yet spent time discussing the actually information to make the
>>decision. This draft addressees several very key issues, including the
>>actually quality comparison of VP8 and H.264. I think that spending any
>>less time than this would result in a situation where the WG was not
>>making an decisions with the information available. Similarly, other
>>people that have written other drafts with similar or different views
>>also need to speak up about how much time they need and that we make
>>sure everyone has time to explain the key information, people have time
>>to ask questions about it, and we can make an informed decisions instead
>>of having to discuss the video codecs at many future meetings.
>>
>>

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 27 Feb 2013 17:50:00 +0000

From: Adam Roach <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, 27 February, 2013 09:45
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [rtcweb] Agenda time request for draft-dbenham-webrtcvideomti

--
Randell Jesup

When trying to figure out what a working group is *supposed* to be doing, I'd generally start with its charter: " the working group will try to avoid encumbered technologies that require royalties or other encumbrances that would prevent such technologies from being easy to use in web browsers."

/a

StW: Oh good.  Let's fall back to H.261, or skip video altogether.  Case closed.

Stephan

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 28 Feb 2013 16:31:12 +0100

Sigh. I thought that after the drubbing draft-dbenham-webrtcvideomti got 
at the previous meeting, the authors would have either improved the 
attempts at quality evaluation or removed them.

It seemed to me that there was rough consensus on the mailing list 
earlier that the quality of the two codecs was close enough that this 
was not going to convince anyone who had already taken a strong position 
based on the IPR issues.

But if we are going to play the video codec quality evaluation game, I 
also have something I want to have on file here.

Google has submitted VP8 as a candidate for standardization in ISO/IEC 
JTC1 SC29 WG11 (better known as MPEG). As part of that submission, we 
submitted a quantitative evaluation of VP8's quality compared to the 
then-current "IVC Test Model", which also included numbers compared to 
the AVC Baseline "anchors" that were part of the project description for 
the IVC effort.

This was contributed to MPEG's January meeting in Geneva; the decision 
at that meeting was to continue the evaluation effort, with new data 
being made available before the next meeting in April.

I'm enclosing the report with the test results; the tests were not done 
by Google; the scripts are available if anyone wants to run them for 
themselves.

                        Harald

---

From: Rob Glidden <[email-masked]>
Date: Thu, 28 Feb 2013 11:08:06 -0800

The full decision is broader, and is in the public meeting resolutions 
(14.1.1 to 14.1.5), below.

Rob

------------------------------------------------------------------------
 From Resolutions, the 103rd SC 29/WG 11 Meeting, 2013-01-21/25, Geneva, 
Switzerland  [SC 29/WG 11 N 13250]
http://www.itscj.ipsj.or.jp/sc29/open/29view/29n13234c.htm

14.1 Internet Video Coding
14.1.1 The Video subgroup recommends approval of the following documents:

*No.*

*Title*

*TBP*

*Available*

*//*

*/Exploration -- Internet Video Coding/*

*//*

*//*

*13353*

*Internet Video Coding Test Model (ITM) v 4.0*

*N*

*13/02/20*

*13354*

*IVC Core Experiment CE1: Overall Codec Testing*

*N*

*13/01/25*

*13355*

*IVC Core Experiment CE2: Improvements of ITM*

*N*

*13/01/25*

14.1.2 The video subgroup would like to point out that an alternative 
technology with potential benefits over the current ITM4 has been 
proposed in M28182 and M28187 for consideration in the IVC 
standardization. A Core Experiment (CE1) was defined for systematic 
testing under comparable conditions. NBs and interested experts are 
encouraged to perform further study of the technology proposed in M28182 
and M28187 regarding the IVC requirements.

14.1.3 The contributors of M28182 and M28187 are asked to provide more 
information about potential restrictions which might prohibit the 
progressing of their technology into an MPEG standard, in case it would 
be considered beneficial from the perspective of IVC development.

14.1.4 WG11 requests its members to review M28182 and M28187 and provide 
suggestions concerning their usability in the IVC process.

14.1.5 WG11 thanks the AUNB for the comment on IVC (M28365). WG11 
considers the actions taken at this meeting to be sufficient in 
responding to AUNB's request.

14.1.6 The video subgroup thanks ITU-T for the viewing equipment that 
was used in the context of evaluating IVC contributions.
------------------------------------------------------------------------

On 2/28/2013 7:31 AM, Harald Alvestrand wrote:
> Sigh. I thought that after the drubbing draft-dbenham-webrtcvideomti 
> got at the previous meeting, the authors would have either improved 
> the attempts at quality evaluation or removed them.
>
> was not going to convince anyone who had already taken a strong 
> position based on the IPR issues.
>
> the AVC Baseline "anchors" that were part of the project description 
> for the IVC effort.
>
> I'm enclosing the report with the test results; the tests were not 
> done by Google; the scripts are available if anyone wants to run them 
> for themselves.
>
>                        Harald
>
>
>

---

From: Gaelle Martin-Cocher <[email-masked]>
Date: Thu, 28 Feb 2013 20:50:27 +0000

Harald,

You may want to clarify that the document you provided is not a report but an input document, and a later revision corrected the source of it.
Further tests and comparisons happen at that meeting (not necessarily very conclusive) and has pointed out by Rob, further tests for the next mpeg meeting were requested. Resolution 14.1.2 is pretty clear .

Sincerely,
Gaelle 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Harald Alvestrand
Sent: Thursday, February 28, 2013 10:31 AM
To: [email-masked]
Subject: [rtcweb] Video codec quality evaluations (Re: Agenda time request for draft-dbenham-webrtcvideomti)

Sigh. I thought that after the drubbing draft-dbenham-webrtcvideomti got at the previous meeting, the authors would have either improved the attempts at quality evaluation or removed them.

It seemed to me that there was rough consensus on the mailing list earlier that the quality of the two codecs was close enough that this was not going to convince anyone who had already taken a strong position based on the IPR issues.

But if we are going to play the video codec quality evaluation game, I also have something I want to have on file here.

Google has submitted VP8 as a candidate for standardization in ISO/IEC
JTC1 SC29 WG11 (better known as MPEG). As part of that submission, we submitted a quantitative evaluation of VP8's quality compared to the then-current "IVC Test Model", which also included numbers compared to the AVC Baseline "anchors" that were part of the project description for the IVC effort.

This was contributed to MPEG's January meeting in Geneva; the decision at that meeting was to continue the evaluation effort, with new data being made available before the next meeting in April.

I'm enclosing the report with the test results; the tests were not done by Google; the scripts are available if anyone wants to run them for themselves.

                        Harald

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 28 Feb 2013 23:18:30 +0100

On 02/28/2013 09:50 PM, Gaelle Martin-Cocher wrote:

Good to hear from you here too, Gaelle!
> You may want to clarify that the document you provided is not a report but an input document, and a later revision corrected the source of it.
Yes, the "Source: AGH on IVC" was both wrong and misspelled!

The source was the three named individuals; it was provided as input to 
the meeting.
> Further tests and comparisons happen at that meeting (not necessarily very conclusive) and has pointed out by Rob, further tests for the next mpeg meeting were requested. Resolution 14.1.2 is prett…

Yep, we're working on it as we speak; they will be ready by the April 
meeting. But not before Orlando!

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 28 Feb 2013 23:21:55 +0100

> (14.1.1 to 14.1.5), below.

Yep, that's the output of the discussion; the file I provided was part 
of the input.

> From Resolutions, the 103rd SC 29/WG 11 Meeting, 2013-01-21/25, 
> Geneva, Switzerland  [SC 29/WG 11 N 13250]
> encouraged to perform further study of the technology proposed in 
> M28182 and M28187 regarding the IVC requirements.
>
> progressing of their technology into an MPEG standard, in case it 
> would be considered beneficial from the perspective of IVC development.
>
> 14.1.4 WG11 requests its members to review M28182 and M28187 and 
> provide suggestions concerning their usability in the IVC process.
>
>> But if we are going to play the video codec quality evaluation game, 
>> I also have something I want to have on file here.
>>
>> Google has submitted VP8 as a candidate for standardization in 
>> ISO/IEC JTC1 SC29 WG11 (better known as MPEG). As part of that 
>> submission, we submitted a quantitative evaluation of VP8's quality 
>> compared to the then-current "IVC Test Model", which also included 
>> numbers compared to the AVC Baseline "anchors" that were part of the 
>> project description for the IVC effort.
>>
>> This was contributed to MPEG's January meeting in Geneva; the 
>> decision at that meeting was to continue the evaluation effort, with 
>> new data being made available before the next meeting in April.
>>

---

From: Gaelle Martin-Cocher <[email-masked]>
Date: Thu, 28 Feb 2013 22:55:32 +0000

Thanks Harald,

:-)
It will be indeed interesting to have the full set of results.
I think the request of MPEG in resolution 14.1.3 is relevant as well to the discussion happening on this list too. 
Would you be in a position to provide an answer in Orlando?
That could help resolve the MTI selection.

Sincerely,
Gaëlle

-----Original Message-----
From: Harald Alvestrand [mailto:[email-masked]] 
Sent: Thursday, February 28, 2013 5:19 PM
To: Gaelle Martin-Cocher
Cc: [email-masked]
Subject: Re: [rtcweb] Video codec quality evaluations (Re: Agenda time request for draft-dbenham-webrtcvideomti)

The source was the three named individuals; it was provided as input to the meeting.

---

From: David Singer <[email-masked]>
Date: Thu, 28 Feb 2013 15:44:52 -0800

> StW: Oh good.  Let's fall back to H.261, or skip video altogether.  Case closed.

I know you were joking, but perhaps we do indeed need something that is simple both to implement and license, as the fall-back codec?  We do, after all, have an end-to-end capability negotiation here -- we're not talking broadcast.  We would like it if we could be sure that that negotiation never comes up empty-handed.  (It would be peachy if it always came up with super quality, at low cost, and with free beer included.)

David Singer
Multimedia and Software Standards, Apple Inc.