Subject: [pm-dir] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 3 Jun 2013 13:48:43 +0000

Hi PMDIR,

As a co-chair of XRBLOCK WG, I would like to request the an RFC 6390 review of draft-ietf-xrblock-rtcp-xr-qoe-08. This document is in WGLC until 6/17.

Thanks and Regards,

Dan

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Tue, 4 Jun 2013 09:22:58 -0400

Dan and pm-dir,

Consider your request made. Do we have a volunteer?

Here's a comment based on my scan of the draft:

Although the quality level is described as QoE,
only estimates of MOS appear in the report blocks.
So rather than relying on the industry's buzzword to
describe the new block, it seems more accurate to simply say: 

RTP Control Protocol (RTCP) Extended Report (XR) Blocks for MOS Metric
                               Reporting

and the related global changes in the text.

And although there is continued discussion and evolution of the definition of QoE,
MOS is just one aspect of QoE - the aspect we have understood for decades.

regards,
Al

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Romascanu, Dan (Dan)
> Sent: Monday, June 03, 2013 9:49 AM
> To: [email-masked]
> Cc: Benoit Claise (bclaise); Gonzalo Camarillo
> Subject: [pm-dir] request for an RFC 6390 review of
> http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> 
> 
> Hi PMDIR,
> 
> As a co-chair of XRBLOCK WG, I would like to request the an RFC 6390
> review of draft-ietf-xrblock-rtcp-xr-qoe-08. This document is in WGLC
> until 6/17.
> 
> Thanks and Regards,
> 
> Dan
> 
> 
> _______________________________________________
> pm-dir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pm-dir

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 4 Jun 2013 13:28:16 +0000

(forwarding also to the xrblock list)

This seems to me like a good comment - Al's proposal makes room in the future for metrics different than the ones reporting MOS values to be used to characterize QoE. I would like to hear the opinion of the authors, and other participants in the WG. 

Thanks and Regards,

Dan

> From: MORTON JR., ALFRED C (AL) [mailto:[email-masked]]
> Sent: Tuesday, June 04, 2013 4:23 PM
> To: Romascanu, Dan (Dan); [email-masked]
> Subject: RE: [pm-dir] request for an RFC 6390 review of
> Although the quality level is described as QoE, only estimates of MOS
> appear in the report blocks.
> So rather than relying on the industry's buzzword to describe the new
> block, it seems more accurate to simply say:
> 
> And although there is continued discussion and evolution of the
> definition of QoE, MOS is just one aspect of QoE - the aspect we have
> understood for decades.
> 
> regards,
> Al
> 
> 
> > From: [email-masked] [mailto:[email-masked]] On
> > Behalf Of Romascanu, Dan (Dan)

---

From: Qin Wu <[email-masked]>
Date: Thu, 6 Jun 2013 05:43:44 +0000

I can understand Al's concern and proposal. However do we have other ways to report QoE in mind besides MoS metric reporting.
MoS metric reporting is widely applicable to a range of RTP application including audio, video and multimedia application.
Not clear to me there will be other alternatives better than MoS metrics reporting?

But I am fine if anybody has no objection to take this proposal.

Regards!
-Qin
-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Romascanu, Dan (Dan)
发送时间: 2013年6月4日 21:28
收件人: MORTON JR., ALFRED C (AL); [email-masked]; 'xrblock'
抄送: Benoit Claise (bclaise)
主题: Re: [xrblock] [pm-dir] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

Thanks and Regards,

Dan

> > review of draft-ietf-xrblock-rtcp-xr-qoe-08. This document is in 
> > WGLC until 6/17.
> >
> > Thanks and Regards,
> >
> > Dan
> >
> >
_______________________________________________
xrblock mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/xrblock

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Wed, 12 Jun 2013 08:14:32 -0400

Dan,

I circulated your request for comments beyond pm-dir, among folks whose work is
referenced and reported in this draft. Several people commented,
and there was an independent request for the change to MOS already
suggested. I have consolidated the three additional comments below.

Please consider these comments during WGLC.

regards,
Al

-=-=-=-=-=-

Since it has just been approved in March 2013, [P.NBAMS-HR] can now be replaced by [P.1202.2].

The list of models referred to in this future RFC should be extended to include G.107.1, 
P.862.1, P.862.2 and P.863.

Important Point concerning MOS scales: 
There is no place in this draft RFC to define the type of MOS scale reported 
according to the terminology defined in P.800.1. 
Here is a simple example. Depending on whether you use or not a measurement result 
for end-to-end delay for the MOS computation (if not, you will use a 0 ms default value), 
the E-model will give you a MOS-CQE or a MOS-LQE (this is terminology from P.800.1). 
Another example is whether you use a Narrow-Band (NB) scale or a mixed Wide Band-NB scale. 
These are only audio examples and there may be similar examples for video, 
but apparently this draft applies to audio MOS alone. In other words, it may not be sufficient
to simply cite the Recommendation used for calculations, there are further details needed
to interpret the MOS value.

-=-=-=-=-=-

> Of MORTON JR., ALFRED C (AL)
> Sent: Tuesday, June 04, 2013 9:23 AM
> Subject: Re: [pm-dir] request for an RFC 6390 review of
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> 
> Dan and pm-dir,
>

---

From: <[email-masked]>
Date: Thu, 13 Jun 2013 08:47:52 +0200

Dear all,

Quality of Experience (QoE), is a subjective measure of a customer's experiences with a service. QoE systems will try to measure metrics that customer will directly perceive as a quality parameter.
In my opinion QoE is a good expression to convey our intention.

Regards

Roland

-----Ursprüngliche Nachricht-----
Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Romascanu, Dan (Dan)
Gesendet: Dienstag, 4. Juni 2013 15:28
An: MORTON JR., ALFRED C (AL); [email-masked]; 'xrblock'
Cc: Benoit Claise (bclaise)
Betreff: Re: [xrblock] [pm-dir] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

Thanks and Regards,

Dan

---

From: Yaakov Stein <[email-masked]>
Date: Thu, 13 Jun 2013 12:45:20 +0000

One should distinguish subjective QoE measures (averaging of measurements on human subjects, e.g., MOS, MUSHRA, BT.500, Apdex)
and objective QoE (i.e., predictions of algorithms that have shown to correlate well with subjective QoE, e.g., PESQ, PEAQ, P.563, PEVQ).

Also note that there are several scales used for QoE type measures.
MOS is from 1 to 5 (and CCR MOS is from -3 to 3). Apdex is from 0 to 1. R-factor is from 0 to 100.
So when retrieving a QoE measure one needs to know what to expect.

Y(J)S

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: 13 June, 2013 09:48
To: [email-masked]; [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [pm-dir] [xrblock] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

Dear all,

Regards

Roland

-----Ursprüngliche Nachricht-----
Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Romascanu, Dan (Dan)
Gesendet: Dienstag, 4. Juni 2013 15:28
An: MORTON JR., ALFRED C (AL); [email-masked]; 'xrblock'
Cc: Benoit Claise (bclaise)
Betreff: Re: [xrblock] [pm-dir] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

Thanks and Regards,

Dan

---

From: Alan Clark <[email-masked]>
Date: Thu, 13 Jun 2013 09:26:05 -0400

I agree that "MOS" is widely used and widely misunderstood. There are many
different MOS ranges 1-5, 1-10, 1-11,....... and a plethora of other QoE
related metrics

ITU P.800.1 helps to an extent for speech MOS - using MOS-[L|C]Q[E|O|S]
where L = Listening, C = Conversational, E = estimated, O = objective and S
= subjective.

Things are getting more complex - with speech/audio there are at least six
different audio bandwidths/ sample rates used, in some cases the MOS/QoE
score is mapped onto different ranges and sometimes onto the same range.
With video we are already seeing 4k resolutions as well as 1080p, 1080i,
720p, 480i......

Yaakov also points out indirectly that there are many organizations that
develop algorithms and standards related to the user perceived quality of
applications (in the broad sense). ITU-T has a long history with speech MOS
and a shorter history with video MOS measurement, ITU-R with video and audio
MOS, and other organizations with data/ application QoE models.

Just a minor point on Yaakov's comments - P.563 obtained good correlation
when averaged over a large set of conditions and speech samples (which is
how ITU tested the algorithm) however performs poorly for individual calls,
with MOS errors of up to +/- 40%.

Regards

Alan

On 6/13/13 8:45 AM, "Yaakov Stein" <[email-masked]> wrote:

> One should distinguish subjective QoE measures (averaging of measurements on
> human subjects, e.g., MOS, MUSHRA, BT.500, Apdex)
> and objective QoE (i.e., predictions of algorithms that have shown to
> correlate well with subjective QoE, e.g., PESQ, PEAQ, P.563, PEVQ).
> 
> Subject: Re: [pm-dir] [xrblock] request for an RFC 6390 review of
> Quality of Experience (QoE), is a subjective measure of a customer's
> experiences with a service. QoE systems will try to measure metrics that
> customer will directly perceive as a quality parameter.
> Betreff: Re: [xrblock] [pm-dir] request for an RFC 6390 review of
> This seems to me like a good comment - Al's proposal makes room in the future
> for metrics different than the ones reporting MOS values to be used to
> characterize QoE. I would like to hear the opinion of the authors, and other
> participants in the WG.
> 
> Thanks and Regards,
> 
> Dan
> 
>

---

From: Qin Wu <[email-masked]>
Date: Fri, 14 Jun 2013 01:01:24 +0000

Hi,all:
It looks to me we are talking about two things.
One is mos type,e.g., listening MoS, Conversational MoS, objective MoS, Subjective MoS.

The other is Mos reference or MoS Scale, e.g., audio with different bandwidth, video with different resolution.

Regarding mos type, I am wondering whether it can be inferred from payload type carried in the QoE XR Block,
If the answer is no, I think we can add one new SDP attribute called "mos type", mapping "mos type" in the SDP 
to CAID used in the packet. Otherwise, we can infer mos type from payload type.

Regarding mos reference or MoS Scale, currently we have already supported 3options using "mosref"attribute,
i.e.,lower resolution, higher resolution and combined lower and higher resolution.
However we may have more than 2 scales,e.g., as Alan pointed out with speech/audio there are at least six
different audio bandwidths/ sample rates used, in order to address this, 
one option is to define more than two option like lower resolution, middle resolution and high resolution and qualify them as typical resolution.
Another option is to remove this 'mosref' and depend on other means to know these information.

Regards!
-Qin
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Alan Clark
Sent: Thursday, June 13, 2013 9:26 PM
To: Yaakov Stein; [email-masked]; Dan (Dan); Al Morton; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [xrblock] [pm-dir] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

Regards

Alan

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Sun, 16 Jun 2013 10:30:21 -0400

It appears this message to pm-dir (June 12) has not yet reached the XRBLOCK list,
but some similar comments have been made. I pass it along now:

==================================================================================

Dan,

regards,
Al

-=-=-=-=-=-

-=-=-=-=-=-

> From: Qin Wu [mailto:[email-masked]]
> Sent: Thursday, June 13, 2013 9:01 PM
> To: Alan Clark; Yaakov Stein; [email-masked]; Dan (Dan); MORTON
> JR., ALFRED C (AL); [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: [xrblock] [pm-dir] request for an RFC 6390 review of
> The other is Mos reference or MoS Scale, e.g., audio with different
> bandwidth, video with different resolution.
> 
> one option is to define more than two option like lower resolution, middle
> resolution and high resolution and qualify them as typical resolution.
> Subject: Re: [xrblock] [pm-dir] request for an RFC 6390 review of
> Of
> > [email-masked]
> > Von: [email-masked] [mailto:[email-masked]] Im
> Auftrag von
> > Romascanu, Dan (Dan)