Subject: [rtcweb] RTP Usage: RTCP XR metrics

From: Magnus Westerlund <[email-masked]>
Date: Mon, 18 Feb 2013 15:07:36 +0100

WG,

In Section 8 of
https://datatracker.ietf.org/doc/draft-ietf-rtcweb-rtp-usage/ there is
an open issue regarding RTCP XR.

The question is if any RTCP XR metrics should be mandated or recommended
to be implemented in WebRTC end-points?

So far there has been some lukewarm interest in metrics, but no clear
proposals on what to include.

As an author I like to either get some discussion going or be able to
remove this open issue. So people, either make a proposal for what to
include or we will remove the TBD and consider the question dealt with.
Please note that there do exist signalling support for negotiating RTCP
XR metrics to use in a particular session.

Submit any proposal no later than the 17th of March.

cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 18 Feb 2013 16:44:34 +0100

draft-alvestrand-rtcweb-stats-registry is the closest I got to actually 
proposing something.
None of the numbers I pulled up there drew definitions from the XR block 
documents.

I support removing this issue at this time; at this stage, I think it is 
best if implementations should add those stats that they find useful, 
and return to the issue of mandating their support later.

On 02/18/2013 03:07 PM, Magnus Westerlund wrote:
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Roni Even" <[email-masked]>
Date: Mon, 18 Feb 2013 19:12:42 +0200

Hi Magnus,
There was a proposal in the past
We have submitted
http://tools.ietf.org/html/draft-huang-rtcweb-monitoring-00 in the past
discussing the RTCP XR metrics for monitoring. 
At the time, a year ago, it was made clear that this is not a priority item.
We can update the draft and submit anew revision.
Thanks
Roni Even

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Magnus Westerlund
Sent: 18 February, 2013 4:08 PM
To: [email-masked]
Cc: Colin Perkins
Subject: [rtcweb] RTP Usage: RTCP XR metrics

WG,

In Section 8 of
https://datatracker.ietf.org/doc/draft-ietf-rtcweb-rtp-usage/ there is an
open issue regarding RTCP XR.

The question is if any RTCP XR metrics should be mandated or recommended to
be implemented in WebRTC end-points?

As an author I like to either get some discussion going or be able to remove
this open issue. So people, either make a proposal for what to include or we
will remove the TBD and consider the question dealt with.
Please note that there do exist signalling support for negotiating RTCP XR
metrics to use in a particular session.

cheers

Magnus Westerlund

---

From: Qin Wu <[email-masked]>
Date: Tue, 19 Feb 2013 13:10:26 +0800

I like to propose to add the following text to Section 8.
For WebRTC application, audio and video are multiplexed onto a single RTP session. 
the basic metrics for transport impairments provided in RTCP SR and RR packets
should be mandatory to be implemented  in WebRTC end-points. However they may 
be not adequate in some cases, e.g., transport impairement reported by RTCP SR and RR
 packets suggesting that quality is not an issue, such as the fact that interarrival jitter is not excessive. 
 However, problems may occur in the service layers leading to poor user experience.

To address the defeciency of RTCP SR and RR packets for performance monitoring, Packet Delay Metrics 
Block, Packet Delay Variation Metrics Block, Burst Gap Loss Metrics Block, Burst Gap Loss Summary Metrics Block,
Burst Gap Discard Metrics Block, Burst Gap Discard Summary Metrics Block, Jitter Buffer Metrics Block, Discard Metrics 
Block, RLE Discard Metrics Block, QoE Metrics Block split from VOIP Metrics Block and all the Metrics Blocks defined in
the RFC3611 other than VOIP Metrics Block are recommended to be implemented in WebRTC end-points if bandwidth is 
not a concern since they are designed for both audio application and video application and can be used by network manager to
troubleshoot network and ensure the quality of real-time application performance.

Regards!
-Qin
----- Original Message ----- 
From: "Magnus Westerlund" <[email-masked]>
To: <[email-masked]>
Cc: "Colin Perkins" <[email-masked]>
Sent: Monday, February 18, 2013 10:07 PM
Subject: [rtcweb] RTP Usage: RTCP XR metrics

WG,

In Section 8 of

cheers

Magnus Westerlund

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 19 Feb 2013 09:34:07 +0000

Leaving to each implementation select 'those stats that they find useful' means no interoperability. Implementations may select different stats, or even if they select the same parameters, in the absence of a common definition of the metrics the results reported by different participants in the same session may be inconsistent. 

Dan

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Harald Alvestrand
> Sent: Monday, February 18, 2013 5:45 PM
> To: [email-masked]
> Subject: Re: [rtcweb] RTP Usage: RTCP XR metrics
> 
> > The question is if any RTCP XR metrics should be mandated or
> > recommended to be implemented in WebRTC end-points?
> >
> > Please note that there do exist signalling support for negotiating
> > RTCP XR metrics to use in a particular session.
> >

---

From: "Schwarz, Albrecht (Albrecht)" <[email-masked]>
Date: Tue, 19 Feb 2013 13:46:44 +0100

Right.
There were RTCP XR based metric types proposed (for RTCWEB), - and they are subject of explicit SDP indication.
Thus, such performance metrics would/could be part of the SDP O/A process in order to address consistency and interoperability in my understanding.

XRBLOCK experts may correct me.

-Albrecht

________________________________________
Von: [email-masked] [[email-masked]] im Auftrag von Romascanu, Dan (Dan) [[email-masked]]
Gesendet: Dienstag, 19. Februar 2013 10:34
An: Harald Alvestrand; [email-masked]
Betreff: Re: [rtcweb] RTP Usage: RTCP XR metrics

Dan

---

From: Colin Perkins <[email-masked]>
Date: Tue, 19 Feb 2013 15:21:54 +0000

I find the meaning of a lowercase "recommended" unclear. Do you mean an RFC 2119-style "RECOMMENDED", or a more general recommendation, akin to an RFC 2119 style "MAY"?

If the aim is that these block types are RECOMMENDED to be implemented, then I think more explanation is needed about why they're important for practically all WebRTC implementations, and under what circumstances it makes sense to omit these blocks. Such strong support for these metrics needs to be justified. 

Alternatively, if you mean that these RTCP XR blocks MAY be implemented, then I see little point in including such a long list, since it doesn't help interoperability. 

Colin

-- 
Colin Perkins
http://csperkins.org/

---

From: Qin Wu <[email-masked]>
Date: Wed, 20 Feb 2013 13:13:06 +0800

Hi, Colin:
My aim is these block types are RECOMMENDED to be implemented, for justification on why they are recommened,
I think we can go to look at four RTCWeb communication requirements:
"
   F5            The browser MUST be able to render good quality
                   audio and video even in the presence of
                   reasonable levels of jitter and packet losses.

   F6            The browser MUST be able to handle high loss and
                   jitter levels in a graceful way.

   F10           The browser MUST support synchronization of
                   audio and video.

   F38          The browser MUST be able to collect statistics,
                   related to the transport of audio and video
                   between peers, needed to estimate quality of
                   service.
"
So my point is if you want to ask browser to ensure good quality for both audio and video in the presence of centain level of jitter and losses,
you should be able to report jitter and losses to help you know what current level of jitter and packet loss is. You may need to collect more detailed
statistics to help you decide what level of jitter and loss you don't want to exceed? PDV metrics block tell you more higher resolution of jitter comparing with using
interarrival jitter provided by RTCP SR/RR.  Statistics Summary Report Block defined in RFC3611 tell you how many packets are lost in a certain sequence number interval.
burst loss rate, gap loss rate defined in the draft-ietf-xrblock-rtcp-xr-summary-stat tell you Proportion of packet lost. Therefore these relevant metrics block SHOULD 
be listed as "RECOMMENDED to be implemented", in my opinion.

In addtion, Browser need to support synchronization of audio and video, however how do you  know a certain browser lost synchronization of audio and video. I think Synchronization delay and offset metrics can tell this. Therefore Synchronization delay and offset metrics, in my opinion, RECOMMENDED to be implemented.

Regards!
-Qin
----- Original Message ----- 
From: "Colin Perkins" <[email-masked]>
To: "Qin Wu" <[email-masked]>
Cc: "Magnus Westerlund" <[email-masked]>; <[email-masked]>
Sent: Tuesday, February 19, 2013 11:21 PM
Subject: Re: [rtcweb] RTP Usage: RTCP XR metrics

Colin

-- 
Colin Perkins
http://csperkins.org/

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 22 Feb 2013 09:33:18 +0100

On 2013-02-20 06:13, Qin Wu wrote:
> Hi, Colin:
> My aim is these block types are RECOMMENDED to be implemented, for
justification on why they are recommened,

> " So my point is if you want to ask browser to ensure good quality
> for both audio and video in the presence of centain level of jitter and losses,
> you should be able to report jitter and losses to help you know what
> current level of jitter and packet loss is. You may need to collect more
> detailed statistics to help you decide what level of jitter and loss you
> don't want to exceed? PDV metrics block tell you more higher resolution of
> jitter comparing with using interarrival jitter provided by RTCP
> SR/RR. Statistics Summary Report Block defined in RFC3611 tell you
> how many packets are lost in a certain sequence number interval.
> burst loss rate, gap loss rate defined in the
> draft-ietf-xrblock-rtcp-xr-summary-stat tell you Proportion of packet
> lost. Therefore these relevant metrics block SHOULD

I don't see the strong requirements of how the WebRTC endpoint
implementation quality requirements (F5, F6 and F10) are strongly linked
to the F38.

One type of arguments is that the peer endpoint being provided with RTCP
XR reports will function better in providing media quality. I don't know
if that is true for the metrics you listed.

A second line of reasoning is around how F38 is addressed. Especially in
the light that the WebRTC endpoints that are browsers will have a stats
API that allows one to get that side of information, assuming that one
has access to influence the JS application and can upload the data to
where it is needed.

For these stats I think the potential motivation for RTCP XR boils down
to this:
1) The peer is not a Browser but has RTCP XR, and one wants to access
its view of the session, thus receiving and gathering RTCP XR reports
enables one to get a better view of the whole RTP session.

2) One want the collection to be in an WebRTC RTP media plane middlebox,
and it should get the endpoints view also, thus RTCP XR is good.

So do people have those reasons, and what information does one really
need in these contexts?

I think some clearer motivations around RTCP XR and each particular
metrics needs to be stated from any proponent.

> In addtion, Browser need to support synchronization of audio and
video, however how do you know a certain browser lost synchronization of
audio and video. I think Synchronization delay and offset metrics can
tell this. Therefore Synchronization delay and offset metrics, in my
opinion, RECOMMENDED to be implemented.

The question is how needs to know that the receiving browser has failed
or selected to not synchronize the media? I don't see that the sender
can do anything different from what they are doing. As I see it all
falls back to knowing what quality level did this session actually
achieve. Especially for synchronization it might be that losing the sync
was better than to increase the audio delay to 1 second as an example.

Cheers

Magnus Westerlund

---

From: Qin Wu <[email-masked]>
Date: Mon, 25 Feb 2013 20:21:37 +0800

----- Original Message ----- 
From: "Magnus Westerlund" <[email-masked]>
To: "Qin Wu" <[email-masked]>
Cc: "Colin Perkins" <[email-masked]>; <[email-masked]>
Sent: Friday, February 22, 2013 4:33 PM
Subject: Re: [rtcweb] RTP Usage: RTCP XR metrics

> has access to influence the JS application and can upload the data to
[Qin]:So you claim the Browser does not need to support RTCP XR in all implementation?

> and it should get the endpoints view also, thus RTCP XR is good.

[Qin]: Sounds good to me. 

> metrics needs to be stated from any proponent.

[Qin] : I think at least transport level metrics carried in RTCP XR need to be supported 
on the peer since they reflect transport impairement in the network affect media quality.
transport level metrics  include Statistics Summary Report Block defined in RFC3611, 
Delay metrics block and PDV metrics block defined in XRBLOCK WG.

For end system metrics, burst gap related metrics, jitter buffer metrics and packet loss 
concealment metrics are RECOMMENDED to be implemented on the peer since these
metrics extend from VOIP summary metrics block defined in RFC3611 and many VOIP
implementations have already support VOIP metrics block reporting.

For application level metrics, at least QoE metrics should be supported on the peer since 
MoS score provide easily understood numetric representation of media quality.

> can do anything different from what they are doing. 

[Qin]: RFC6051 defines three methods to reduce  the
  possible synchronisation delay. Two methods can be used by sender.

>As I see it all