Subject: Re: [pm-dir] Request for an RFC 6369 review of draft-ietf-xrblock-rtcp-xr-summary-stat-06

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Sat, 2 Feb 2013 17:50:36 -0500

Benoit,

Since there is a short term deadline 
(the Last Call has ended on Feb 1) 
I'll provide the PM Dir review.

Al

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Since the draft describes reporting formats and (apparently) does not 
intend to define new metrics, much of 6390 guidance is out of scope.
However, see the comments on section 4.1.2 below regarding metric definitions.

Comments:

Section 1.1 of the Intro says:

   This draft defines three new block types to augment those defined in
   [RFC3611] for use in a range of RTP applications.

ok so far, but the next paragraph is hard to decipher, perhaps a simple
numbered list of the new blocks would help:

    1. Burst/Gap Loss Summary Statistics Metrics Block
    2. Burst/Gap Discard Summary Statistics Metrics Block
    3. Frame Impairment Statistics Summary Metrics Block

back to the existing text:
   The first two block types support the reporting of burst gap loss/
   discard summary statistics including packet loss/discard proportion,
   mean and variance and belong to the class of transport-related end
   system metrics defined in [RFC6792].  These two blocks are intended
   to be used in conjunction with information from the Burst Gap Loss
   Metrics Block or Burst Gap Discard Metrics Block, and on which these
   two block therefore depend.  
add a reference to [RFC3611] for these two blocks.

   The metrics in the Burst Gap Loss block
   or Burst Gap Discard Metrics Block can be used independently of the
   metrics defined in the first two blocks.

That's obvious, because anyone implementing RFC3611 is already doing it.

The authors appear to have avoided using "video" as an adjective when 
referring to video frames throughout the memo. For example:

   The third block supports the reporting of detailed statistics for
   each frame type, including the number of frames received, lost and
   discarded of each frame type in the Group of Pictures (GOP) and
   additional data allowing the calculation of statistical parameters
   (e.g.,the proportion of each frame type impaired by packet loss and
   discard).  The metrics defined in this block belong to the class of
   application layer metrics defined in [RFC6792].

These are all video frames above, no?  At least say so once.

Section 2.1, on Terminology, says

    Picture Type

      Picture Types used in the different video algorithms are composed
      of the Key frame and Derived frames.  The Key frame is also called
      a reference frame and used as a reference for predicting other
      pictures.  It is coded without prediction from other pictures.
      Derived frames are derived from a Key frame using a prediction
      algorithm.

At first this seems quite generic, but the definition may fail when one slice
of the video frame is independently coded, and other slices are predictively
coded (this has been done to smooth the video rate). Is any video frame that
includes independent or reference coding a Key frame?

Section 3.1.2 says:
   Burst Loss Rate: 16 bits

      The fraction of packets lost during bursts since the beginning of
      reception, expressed as a fixed point number with the binary point
      at the left edge of the field.  This value is calculated by
      dividing Packets Loss in Bursts by Total Packets expected in
      Bursts as follows:
          Packets Loss in Bursts / Total Packets expected in Bursts

How is a burst loss ratio = 1.0 reported?  All the digits are to the 
right of the decimal place.

   Burst Duration Mean:16bits

      The mean burst duration is obtained as the quotient:

      mean = Sum of Burst Durations / Number of Bursts

How is Divide by Zero handled?

   Burst Duration Variance:16bits

      The variance of the burst duration is obtained using the standard
      result:

      var = ( Sum of Squares of Burst Durations - Number of Bursts *
      mean^2 ) / (Number of Bursts - 1)

How is Divide by Zero handled? (e.g., Number of Bursts = 1)

Section 4.1.2 says

   Number of full frames lost (lost_full_frames): 32 bits

      If one frame is completely lost, this frame is regarded as one
      lost full frame.  The lost_full_frames is equivalent to the number
      of full frames lost in the above sequence number interval.

Is this a metric definition, fully lost video frames? 
How is this event detected so it can be counted?
Is there a reference for this metric available?

   Number of partial frames lost (lost_partial_frames): 32 bits

      If one frame is partially lost, this frame is regarded as one lost
      fractional frame.  The value of the lost_partial_frames field is
      equivalent to the number of partial frames lost in the above
      sequence number interval.

Is this a metric definition, partially lost video frames? 
How is this event detected so it can be counted?
Is there a reference for this metric available?

-=-=-=-=-=-=-

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Benoit Claise
> Sent: Sunday, January 27, 2013 2:09 PM
> To: [email-masked]
> Cc: [email-masked]; draft-ietf-xrblock-rtcp-xr-summary-
> [email-masked]; Gonzalo Camarillo
> Subject: Re: [pm-dir] Request for an RFC 6369 review of draft-ietf-
> xrblock-rtcp-xr-summary-stat-06
> 
> Dear all,
> 
> Here is an official request for the RFC 6390 review.
> Who would like to volunteer?
> 
> Regards, Benoit
> > Hi Benoit,
> >
> > as suggested by Dan in his PROTO writeup, I would like to request the
> > performance metrics directorate to perform an RFC 6390 review of the
> > following draft as part of its IETF LC:
> >
> > https://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcp-xr-summary-
> stat/
> >
> > Could you please arrange such a review?
> >
> > Thanks,
> >
> > Gonzalo
> >
> >
> 
> _______________________________________________
> pm-dir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pm-dir

---

From: Qin Wu <[email-masked]>
Date: Mon, 4 Feb 2013 16:40:31 +0800

Hi,Al:
Thank for your valuable review. We will address your comments in the update.
Please see my reply below.
----- Original Message ----- 
From: "MORTON JR., ALFRED (AL)" <[email-masked]>
To: "Benoit Claise" <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; "Gonzalo Camarillo" <[email-masked]>
Sent: Sunday, February 03, 2013 6:50 AM
Subject: RE: [pm-dir] Request for an RFC 6369 review ofdraft-ietf-xrblock-rtcp-xr-summary-stat-06

Benoit,

Al

Comments:

[Qin]: Good suggestion. Thanks.

[Qin]: Okay and will add that.

[Qin]: Agree.

[Qin]: Agree and  will fix this. 

    Picture Type

[Qin]: One is independently coded and the other is predictively coded, that is how the key frame 
distinct fromthe  derived frame. One example of Key frame is I frame which is independently coded
reference frame. Is there any exceptional case?

Section 3.1.2 says:
   Burst Loss Rate: 16 bits

[Qin]: Good comment, I think we can use 15bits of 16-bit Burst gap Rate
to represent the value of burst loss rate =1.0. The leftmost bit is used to indicate measurement
is unavailable. The proposed change is as follows:
OLD TEXT:
"
   Burst Loss Rate: 16 bits

"
NEW TEXT:
"
   Burst Loss Rate: 16 bits

      Bursts, multiplying the result of the division by 32768 , and taking the integer part as follows:

      Packets Loss in Bursts / Total Packets expected in Bursts
      If the measurement is unavailable, the value 0x8000 MUST be
      reported.
"

[Qin]: Good comment, we will use 0xFFFF to indicate measurement unavailable.

[Qin]: Good comment, we will use 0xFFFF to indicate measurement unavailable.

Section 4.1.2 says

[Qin]: Yes.

[Qin]: 
A frame is either split across multiple packets or carried in only one packet,
Suppose a frame is split across multiple packets,
if we know how many packets comprise a frame(e.g., by looking at sequence 
number in the RTP header and continue counter in the TS header), we will see whether
a frame is full lost frame or partially lost frame.
If some packets of a frame is lost, we will regard it as partial lost frame,
If all packets of frame is lost, this frame will be regarded as full lost frame.

[Qin]: Note aware of it. 
-=-=-=-=-=-=-

---

From: Benoit Claise <[email-masked]>
Date: Fri, 15 Feb 2013 00:18:30 +0100

Dear authors,

I'm reviewing version 8 for the IESG call next week.
> I'll provide the PM Dir review.
This is the key question.
Do you define new metrics?
When I see the following (to take just one example):

      Burst Loss Rate: 16 bits

           The fraction of packets lost during bursts since the beginning of
           reception, expressed as a fixed point number with the binary point
           at the left edge of the field.  This value is calculated by
           dividing Packets Loss in Bursts by Total Packets expected in
           Bursts, multiplying the result of the division by 7FFF, with the
           maximum value 7FFF, and taking the integer part as follows:

... it seems to me that you define new metrics, and that therefore the 
RFC 6390 template should be applied.
What do I miss?

Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Wed, 20 Feb 2013 00:58:53 +0100

Dear pm-dir,

[reducing the list to pm-dir and Gonzallo, XRBLOCK AD]
I filed a DISCUSS on this document.
See 
https://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcp-xr-summary-stat/ballot/#benoit-claise 
for the details.
We should give the right guidelines to performance metrics related draft 
authors.
Maybe we should even mention 
http://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcp-xr-pdv/ballot/#benoit-claise 
in our WIKI?

Regards, Benoit
>>> From:[email-masked]  [mailto:[email-masked]] On Behalf
>>> Of Benoit Claise
>>> To:[email-masked]
>>> Cc:[email-masked]; draft-ietf-xrblock-rtcp-xr-summary-

---

From: Gonzalo Camarillo <[email-masked]>
Date: Wed, 20 Feb 2013 10:23:00 +0200

Hi Benoit,

thanks for following this up. I have already told Dan, who is this
document's shepherd, to get involved in the discussions so that we can
move forward.

Cheers,

Gonzalo