Subject: [rtcweb] Stephan Wenger's Video Document Comments

From: Adam Roach <[email-masked]>
Date: Thu, 12 Feb 2015 15:32:49 -0600

Splitting this off from the thread it was posted on for the sake of 
cleanliness.

Thanks, Stefan, for the very useful feedback. I've incorporated it into 
the upcoming -04 version of the document as described below.

On 11/25/14 23:16, Stephan Wenger wrote:
> 1. Section 3: sRGB is fine.

Done.

> 2. Section 3: Add something like dynamic frame rate / frame interval based
> on user locale, available bandwidth, and so on.  For example, in low light
> conditions, it almost never makes sense to run a full 30 fps camera when
> your transport has only bandwidth for 15 fps.

I think I follow what you're saying, but not 100% sure. What I've added is:

* Dynamic frame rate for video capture based on actual encoding in use
   (e.g., if encoding at 15 fps due to bandwidth constraints, the camera 
will
   ideally capture at 15 fps rather than a higher rate).

> 3. Section 3: Possibly missing: mention that the video scan pattern for
> both VP8 and H.264 is YCrCb 4:2:0.

Added: "Unless specified otherwise by the SDP or codec, the video scan 
pattern for
video codecs is Y'CbCr 4:2:0".

> 4. Section 3: Possibly missing: the screen content section should mention
> that the scan format of video (YCrCb 4:2:0) is known to be suboptimal for
> the representation of screen content as produced by systems at the time of
> publication, which generally use at least 24 bits per sample RGB.  Perhaps
> a forward looking statement: ³In the future, it may become advisable to
> use video codecs optimized for screen content (scan pattern, color space,
> signal characteristics) for the representation of this type of content.²

Added:

Note that the default video scan format (Y'CrCb 4:2:0) is known to be
less than optimal for the representation of screen content produced by
most systems in use at the time of this document's publication, which
generally use RGB with at least 24 bits per sample. In the future, it
may be advisable to use video codecs optimized for screen content for the
representation of this type of content.

> 5. Section 5 second and third paragraphs: Mention Constrained Baseline
> Profile here.   Otherwise, section 5 seems fine to me.  (I know that
> constrained baseline is mentioned as a MUST later, but H.264 in isolation
> is insufficient even in this overview part of the document).

Done.

> 6. Section 6, 3rd para, 1st and 2nd line, nit: "are MUST²

Fixed.

> 5. Section 6.2, profile_level_id, suggest to change the receiving side to
> a MUST.  The decoder interprets the value anyway as presented in-band.
> What¹s the point that a receiver accepts a bitstream (without checking
> profile_level_id), and the croaks because the decoder cannot handle the
> bitstream complexity as communicated in-band?

done.

> 6. Section 6.2, max_mbps..., nit: par ameters (1st and second line)

That's a very strange tooling issue (it's fine in the source). I'm going 
to see if I can untangle what's going on here. Thanks for catching it.

>
> Normative reference to H.264: depending on the timing, it may be better to
> reference H.264v9.  That version is currently in ³pre-published² state and
> not freely downloadable, but should be generally available rather sooner
> than later (and almost certainly before we are through with this draft).

Updated the reference. Thanks for catching this.

> 7. Section 6.2, sprop_parameter_sets: For avoidance of doubt, I would
> rephrase that as ³this parameter MUST NOT be present².  It is possible to
> have that parameter and later change parameter sets in-band, and that is a
> condition I think you want to avoid.

That seems like a good point. I've taken the suggestion as:

: H.264 allows sequence and picture information to be sent both in-band, and
   out-of-band.  WebRTC implementations MUST signal this information 
in-band.
   This means that WebRTC implementations MUST NOT include this parameter in
   the SDP they generate.

> 8. Section 6.2, Possibly missing: for H.264, should we specify a pixel
> aspect ratio?  VP8 uses square pixels, right?  Perhaps follow that lead?

Added: "Unless otherwise signaled, implementations that use H.264 MUST 
encode and decode pixels with a implied 1:1 (square) aspect ratio."

> 9. Section 7: point to the payload formats also.  H.264 allows for active
> content in SEI messages... and the payload formats contain the necessary
> caveats.

Added:

Implementors making use of H.264 are also advised to take careful note 
of the
"Security Considerations" section of {{RFC6184}}, paying special regard 
to the
normative requirement pertaining to SEI messages.

/a

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 13 Feb 2015 09:47:48 +0000

Hi,
We seem to be in agreement.  There is just one point where I have additional input; see below (including suggested text).
Stephan

From: Adam Roach <[email-masked]<mailto:[email-masked]>>
Date: Thursday, February 12, 2015 at 22:32
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Stephan Wenger <[email-masked]<mailto:[email-masked]>>
Subject: Stephan Wenger's Video Document Comments

Splitting this off from the thread it was posted on for the sake of cleanliness.

Thanks, Stefan, for the very useful feedback. I've incorporated it into the upcoming -04 version of the document as described below.

[...]

* Dynamic frame rate for video capture based on actual encoding in use
  (e.g., if encoding at 15 fps due to bandwidth constraints, the camera will
  ideally capture at 15 fps rather than a higher rate).

Transmission bandwidth (which comes to play only after encoding, and where the encoder generally has many choices in addition to dealing down the frame rate) is just one reason to go down with the capture frame rate.  Application needs/settings is another and probably a more common one.  Yet another is light conditions: in low light conditions, one is better off to sample at low frame rate, for two reasons: first, the human eye is not particularly good in observing fast movement in low light, so why bother capturing/coding it?  And second, a camera sensor creates more noise at low light conditions, which can be mitigated by allowing more photons to hit the sensor element in question over time.
Also, capture frame rate can be dictated not only by the codec (as a result of bandwidth starvation), but also by the environment (light/applications setting).  Your text suggests that the frame rate adjustment trickles down from transmission needs, but that's not the only source (and perhaps not even the most relevant one).

Suggested text:

  *   Dynamic frame rate for video capture based on actual encoding in use (e.g., if encoding at 15 fps due to bandwidth constraints, low light conditions, or application settings, the camera will ideally capture at 15 fps rather than a higher rate).