Subject: Re: [pm-dir] request for an early PMDIR review

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Mon, 26 Aug 2013 10:12:51 -0400

Hi Dan,

As promised, here's an early pm-dir review of the draft:
https://datatracker.ietf.org/doc/draft-huang-xrblock-rtcweb-rtcp-xr-metrics/

Since the purpose of this draft is to make recommendations 
for measurements to satisfy the WebRTC requirement cited in the draft:

   "F38    The browser MUST be able to collect statistics, related to
   the transport of audio and video between peers, needed to estimate
   quality of service."

"Statistics" seemed a bit vague to me, especially in a requirement.
One supporting reference is:

   [RTCWEB-STAT] Alvestrand, H., "A Registry for WebRTC statistics
                 identifiers", September 24, 2012.

I took a look at Harald's draft. "RTCWeb Media Statistics" are indeed the 
result of some performance measurement, and the registry he proposes 
appears to assume that all needed metric definition details can be
found in a specification or supplied by the expert who prepares the
registry entry. There would likely be overlap between the
"RTCWeb Media Statistics" registry and other registries containing
performance metrics.

There's a *missing* aspect of the WebRTC requirement F38:
Specification of the *minimum* set of metrics that MUST be provided
to accomplish the stated goal: estimate quality of service
(and not quality of experience!).

It seems to me that draft-huang-xrblock-rtcweb-rtcp-xr-metrics
*could* be revised to fulfill that role. Otherwise, the intent and
goal of F38 will never be realized because implementers can make
different choices of statistics, and possibly include a set which
is insufficient. In addition, the draft currently takes the position 
that some metrics are needed for network performance diagnosis.
This is a valuable set of metrics too, but they need to be distinguished
from the minimum set of metrics needed to estimate QoS and 
justified through explanation of their diagnostic power 
(as has been done in some cases already in the draft).
Again, emphasis should be on a *minimum* set of diagnostic metrics,
and which applications and common circumstances.

Some more specific comments follow:

3.2.1 Retransmitted Packet Count Metric
I'm surprised to see this suggested, especially as the first in the memo.
In true Real-time conversational applications, there's little opportunity
for retransmission to be effective. The draft says:
   In low delay networks with low
   loss rates, retransmissions have great value without incurring
   additional complexity.
If the loss ratio is low, it seems retransmission is not very valuable,
and low delay networks are a very restricted set. I don't see what 
additional diagnostic value this metric has, beyond the other loss metrics.
How widely is the NACK capability deployed and used?
Note: there could be link-layer retransmissions below RTP-layer,
but these are not visible above the link layer, except as 
reordered packets.

3.2.3 Loss, Discard and Duplicate Packet Count Metric
I agree that Loss and Discard are valuable for QoS and diagnostic
purposes, but in my experience Duplicate packets are rare and have
little effect on QoS (they are usually discarded silently, but they
should not appear in the discard counts).

3.2.5 Run Length Encoded Metrics for Loss, Discard and Post-repair

   Run-length encoding uses a bit vector to encode information about the
   packet. Each bit in the vector represents a packet and depending on
   the signaled metric it defines if the packet was lost, duplicated,
   discarded, or repaired. An endpoint typically uses the run length
   encoding to accurately communicate the status of each packet in the
   interval to the other endpoint.
This seems like an enormous amount of data to convey between
the receiver and sender, a fairly complete record of transmission.
It might be a diagnostic capability turned-on in some specific 
troubleshooting cases, but I don't see RLE as part of the 
QoS estimation metrics set.

It's fine to include Jitter and De-jitter buffer metrics in the QoS 
estimation set. Others could be discussed, but it's probably good to
stop here for now.

hope this helps,
Al
PS I'm switching to other topics, and will probably delay any responses
on this thread for a couple of weeks.

> -----Original Message-----
> From: Romascanu, Dan (Dan) [mailto:[email-masked]]
> Sent: Thursday, August 01, 2013 3:01 PM
> To: MORTON JR., ALFRED C (AL)
> Cc: Huangyihong (Rachel); Varun Singh; Roni Even; Shida Schubert
> Subject: request for an early PMDIR review
> 
> Hi Al,
> 
> Following our discussion earlier today, I would kindly ask for an early
> PM-DIR review of https://datatracker.ietf.org/doc/draft-huang-xrblock-
> rtcweb-rtcp-xr-metrics/.
> 
> Thanks and Regards,
> 
> Dan
> 
> 
>

---

From: Varun Singh <[email-masked]>
Date: Mon, 26 Aug 2013 17:34:56 +0300

Hi Al,

Thanks for the review. Comments inline.

Cheers,
Varun

> "RTCWeb Media Statistics" registry and other registries containing

Our intention with the upcoming versions of the draft was to add to 
the registry proposed by Harald.

> and which applications and common circumstances.

I agree we should define a minimum set of metrics for the next version, 
the current version of the draft was mainly to organize all the available 
metrics/measures and to facilitate discussion. Would your suggestion 
be to: just define the minimum set or keep the current discussion and then 
define the minimum set?

> but these are not visible above the link layer, except as 

Short answer, Chrome uses NACK or PLI (packet loss indication) extensively. 
My measurements show that there are cases in low delay where the actual packet loss 
is 10-20% and by using retx they are able to bring it down to <1%, these measurements 
are basically within northern europe (end-to-end delay up to 50ms, AFAIK, the chrome
implementation tries to meet the 150ms one-way delay guarantees).

They also use FEC, which can sometimes be 10-20% of the overall bit rate. We try 
to capture this in the post-repair metric in 3.2.5. Perhaps, we may have to define
a new Post-repair metric which is not RLE, but a measure of number of packets 
recovered by using FEC.

> should not appear in the discard counts).
Ok, remove duplicate and keep the discard and loss metric?

> QoS estimation metrics set.

I agree, RLE may be an overkill for the general monitoring.

> estimation set. Others could be discussed, but it's probably good to

Thanks again for the comments.

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Mon, 26 Aug 2013 10:53:26 -0400

Hi Varun,
my quick replies to your questions:

> From: Varun Singh [mailto:[email-masked]]
...
> be to: just define the minimum set or keep the current discussion and then

I suggest to just define the minimum set for QoS estimation and 
another, overlapping set of metrics with diagnostic properties
in this draft.  The survey of possible metrics is less valuable, IMO.

> > 3.2.1 Retransmitted Packet Count Metric
...
> implementation tries to meet the 150ms one-way delay guarantees).

Ok, I wouldn't have called 10-20% loss "low", and that's significant improvement.
Incidentally, the 150ms one-way delay objective from G.114 includes
the hosts and codecs, basically user to user delay.

> a new Post-repair metric which is not RLE, but a measure of number of

Yes, that's very valuable to know: there is loss, but within the FEC capabilities
to repair so there's little user impact beyond the FEC overhead and delay
due to block processing, so the trade-off is a good one.

> Ok, remove duplicate and keep the discard and loss metric?

I think so.

> > 3.2.5 Run Length Encoded Metrics for Loss, Discard and Post-repair
...
> I agree, RLE may be an overkill for the general monitoring.
Good, the really valuable info is about the repaired packets.

glad this was useful,
Al

---

From: "Huangyihong (Rachel)" <[email-masked]>
Date: Tue, 27 Aug 2013 03:24:26 +0000

Hi Al,

Thank you for these valuable comments. Please see some additional replies inline.

Best regards,
Rachel

-----Original Message-----
From: MORTON JR., ALFRED C (AL) [mailto:[email-masked]] 
Sent: Monday, August 26, 2013 10:13 PM
To: Romascanu, Dan (Dan); [email-masked]
Cc: Huangyihong (Rachel); Varun Singh; Roni Even; Shida Schubert
Subject: RE: request for an early PMDIR review

Hi Dan,

[Rachel]: Are you suggestion that we should focus the metrics evaluating QoS while remove the QoE related ones? Actually, the reason why we consider QoE metrics (such as frame impairment metrics) is because they are useful to diagnostic why media quality degrades. We think it's important because people care about the quality. But I agree with you that they're not common enough to all applications. It's fine to take them out.

[Rachel]: The metrics are not ordered by their importance, instead, they are listed from transport level to application level. I'm sorry it confused you. We can adjust the order in the next version. I have no additional comments other than Varun. 

[Rachel]: Okay. Agree.

[Rachel]: Okay.

hope this helps,
Al