Subject: Re: [codec] Semi-formal codec quality testing

From: Michael Knappe <[email-masked]>
Date: Thu, 2 Sep 2010 06:25:32 -0700

Roman,

Requirements are hopefully quite reasonable:

A) a quiet ‘NC25’ listening environment of approx 35 dBA or less ambient noise. Not a necessity to be an acoustically treated room for reverberation control given that listening will done over headphones, but it doesn’t hurt either. This can be an existing room, but an ISOBOOTH will also work great.
B) a computer that can run a MUSHRA assessment tool (like RateIt or MUSHRAM)
C) a high quality D/A convertor connected via USB or Firewire to the assessment computer (e.g.  Benchmark DAC,  Metric Halo ULN-2, Apogee miniDAC)
D) high quality headphone amp and playback level calibration (many good convertors like the Metric Halo also have a good quality headphone amp built in) - playback calibration can be set via an Etymotic in-ear microphone – we’ll decide on a standard calibration scheme shortly
E) high quality headphones – a decent set of dynamic headphones with flat frequency response and reasonably detailed (e.g. AKG 240DF, Sennheiser HD600), Stax electrostatics also great.

Let me know if you have any questions at all,

Cheers,

Mike

On 8/30/10 1:09 PM, "Roman Shpount" <[email-masked]> wrote:

Can you provide a brief description of test site requirements that are needed to participate in this? What do you consider a reasonable playback environment? Would we need regular or trained listeners?
_____________
Roman Shpount

On Fri, Aug 27, 2010 at 6:07 PM, Michael Knappe <[email-masked]> wrote:
As discussed at IETF78, we’d like to see a number of organizations / corporations donate time and effort to participate in ‘semi-formal’ audio quality testing of the output of this working group. This would not be a paid activity, and would require each organization to provide a suitable listening environment and playback equipment chain for their testing. My estimate is that a reasonable controlled playback environment can be put together for approximately 5-10K, but many organizations interested in this activity are likely to already possess the bulk of the setup.

If you’re interested, please email back so I can get an estimate of the number of independent test sites we’ll eventually have, and we’ll begin the process of collaborating on the mailing list to put together source material, test cases, etc.

Cheers,

Mike
_______________________________________________
codec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/codec

---

From: Michael Knappe <[email-masked]>
Date: Thu, 2 Sep 2010 06:29:22 -0700

Forgot to add that a demographic cross-section of regular listeners is preferred, but will also be useful to run tests with expert listeners as well.

Cheers,

Mike

On 9/2/10 6:25 AM, "Michael Knappe" <[email-masked]> wrote:

Roman,

Cheers,

Mike

Cheers,

Mike

---

From: Anisse Taleb <[email-masked]>
Date: Thu, 02 Sep 2010 16:19:47 +0200

Dear Michael,

I wonder what do you mean by Regular listeners ? Note that according to
BS.1534 (Mushra), quoted section 4. for conveninece :

" Although the MUSHRA test method is not intended to be applied to small
impairments, it is still
recommended that experienced listeners should be used. These listeners
should have experience in
listening to sound in a critical way. Such listeners will give a more
reliable result more quickly than
non-experienced listeners. It is also important to note that most
non-experienced listeners tend to
become more sensitive to the various types of artefacts after frequent
exposure."

Kind regards,
/Anisse

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Michael Knappe
> Sent: den 2 september 2010 15:29
> To: Roman Shpount
> Cc: [email-masked]
> Subject: Re: [codec] Semi-formal codec quality testing
> 
> Forgot to add that a demographic cross-section of regular listeners is
> preferred, but will also be useful to run tests with expert listeners
> as well.
> 
> Cheers,
> 
> Mike
> 
> 
> A) a quiet 'NC25' listening environment of approx 35 dBA or less
> ambient noise. Not a necessity to be an acoustically treated room for
> reverberation control given that listening will done over headphones,
> but it doesn't hurt either. This can be an existing room, but an
> ISOBOOTH will also work great.
> C) a high quality D/A convertor connected via USB or Firewire to the
> assessment computer (e.g.  Benchmark DAC,  Metric Halo ULN-2, Apogee
> miniDAC)
> D) high quality headphone amp and playback level calibration (many good
> convertors like the Metric Halo also have a good quality headphone amp
> built in) - playback calibration can be set via an Etymotic in-ear
> microphone - we'll decide on a standard calibration scheme shortly
> E) high quality headphones - a decent set of dynamic headphones with
> flat frequency response and reasonably detailed (e.g. AKG 240DF,
> Sennheiser HD600), Stax electrostatics also great.
> 
> Can you provide a brief description of test site requirements that are
> needed to participate in this? What do you consider a reasonable
> playback environment? Would we need regular or trained listeners?
> _____________
> Roman Shpount
> 
> 
> As discussed at IETF78, we'd like to see a number of organizations /
> corporations donate time and effort to participate in 'semi-formal'
> audio quality testing of the output of this working group. This would
> not be a paid activity, and would require each organization to provide
> a suitable listening environment and playback equipment chain for their
> testing. My estimate is that a reasonable controlled playback
> environment can be put together for approximately 5-10K, but many
> organizations interested in this activity are likely to already possess
> the bulk of the setup.
> 
> If you're interested, please email back so I can get an estimate of the
> number of independent test sites we'll eventually have, and we'll begin
> the process of collaborating on the mailing list to put together source
> material, test cases, etc.
> 
> Cheers,
> 
> Mike

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 02 Sep 2010 10:45:25 -0400

Hi,

 > Such listeners will give a more reliable result more quickly than
 > non-experienced listeners.

 From my experience, this shouldn't be too much of a problem because unless 
two codecs are very close in quality, MUSHRA tends to produce statistically 
significant results pretty quickly, even with naive listeners. This is what 
happened with the informal listening test we did on the hybrid codec.

That being said, it surely wouldn't hurt have some experienced listeners to 
confirm any results we get with naive listeners.

Cheers,

	Jean-Marc

On 10-09-02 10:19 AM, Anisse Taleb wrote:
>> On 9/2/10 6:25 AM, "Michael Knappe"<[email-masked]>  wrote:
>>
>> Roman,
>>
>> On 8/30/10 1:09 PM, "Roman Shpount"<[email-masked]>  wrote:
>>
>> On Fri, Aug 27, 2010 at 6:07 PM, Michael Knappe<[email-masked]>
>> wrote: