Subject: [rtcweb] Finishing up the Video Codec document

From: Adam Roach <[email-masked]>
Date: Tue, 25 Nov 2014 17:33:47 -0600

I've updated the draft-ietf-rtcweb-video based on the minutes from 
Hawaii. Now that we have a clear path to success, I'd like to get the 
document finalized and published as quickly as possible. This means I 
need your feedback on the remaining open issues (1, 4, and 5, below). If 
you are CCed on this mail, it's because you took an action item in 
Hawaii, and I'm waiting to hear back from you.

New version: http://datatracker.ietf.org/doc/draft-ietf-rtcweb-video/
Diff: https://tools.ietf.org/rfcdiff?url2=draft-ietf-rtcweb-video-03.txt
____

Action item for Harald Alvestrand from the minutes:
> Mo Zanaty: An IANA registered header with reference to 3GPP document. 
> Codec specific mechanism way of doing this.
> Harald: Will send some suggestions on doing this.

Action item for Bernard Aboba and Stephan Wenger: collect list of H.264 
SEI messages we want to call out as MUST and/or SHOULD.
____

Open Issue 1: If you don't want sRGB to be the default color space, say 
something now.

Open Issue 2 (closed): Took "screen source video metadata" issue out of 
the document, as this is an RTP issue, not a codec issue.

Open Issue 3 (closed): New text in "stream orientation" section calling 
out various MUST/SHOULD/MAY levels, based on what I took away from the 
mic in Hawaii. If you care about this, double-check that these are what 
you expect them to be.

Open Issue 4: In Toronto, it was suggested that we don't need to specify 
support for "bilinear" and "none" filter styles, since they're already 
required. I'm happy to remove the statement if someone can point me to 
where this is otherwise mandated. If no one comes forward with a 
citation, I'm going to close this open issue as harmless.

Open Issue 5: Waiting on feedback from Bernard and Stefan, as detailed 
above.

Open Issue 6 (closed): Please see the text in section 5 of the new 
document, which tracks the text from the Novel Proposal email pretty 
closely.

/a

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 26 Nov 2014 05:16:55 +0000

Hi,
below a few generic comments (not directly related to my action item).
Stephan

1. Section 3: sRGB is fine.

2. Section 3: Add something like dynamic frame rate / frame interval based
on user locale, available bandwidth, and so on.  For example, in low light
conditions, it almost never makes sense to run a full 30 fps camera when
your transport has only bandwidth for 15 fps.

3. Section 3: Possibly missing: mention that the video scan pattern for
both VP8 and H.264 is YCrCb 4:2:0.

4. Section 3: Possibly missing: the screen content section should mention
that the scan format of video (YCrCb 4:2:0) is known to be suboptimal for
the representation of screen content as produced by systems at the time of
publication, which generally use at least 24 bits per sample RGB.  Perhaps
a forward looking statement: ³In the future, it may become advisable to
use video codecs optimized for screen content (scan pattern, color space,
signal characteristics) for the representation of this type of content.²

5. Section 5 second and third paragraphs: Mention Constrained Baseline
Profile here.   Otherwise, section 5 seems fine to me.  (I know that
constrained baseline is mentioned as a MUST later, but H.264 in isolation
is insufficient even in this overview part of the document).

6. Section 6, 3rd para, 1st and 2nd line, nit: "are MUST²

5. Section 6.2, profile_level_id, suggest to change the receiving side to
a MUST.  The decoder interprets the value anyway as presented in-band.
What¹s the point that a receiver accepts a bitstream (without checking
profile_level_id), and the croaks because the decoder cannot handle the
bitstream complexity as communicated in-band?

6. Section 6.2, max_mbps..., nit: par ameters (1st and second line)

Normative reference to H.264: depending on the timing, it may be better to
reference H.264v9.  That version is currently in ³pre-published² state and
not freely downloadable, but should be generally available rather sooner
than later (and almost certainly before we are through with this draft).

7. Section 6.2, sprop_parameter_sets: For avoidance of doubt, I would
rephrase that as ³this parameter MUST NOT be present².  It is possible to
have that parameter and later change parameter sets in-band, and that is a
condition I think you want to avoid.  (btw. my heart is bleeding.  Some of
you know why.  The rest: keep guessing :-)

8. Section 6.2, Possibly missing: for H.264, should we specify a pixel
aspect ratio?  VP8 uses square pixels, right?  Perhaps follow that lead?

9. Section 7: point to the payload formats also.  H.264 allows for active
content in SEI messages... and the payload formats contain the necessary
caveats.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 25 Nov 2014 21:25:38 -0800

> both VP8 and H.264 is YCrCb 4:2:0.

You mean Y'CbCr, no?

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 26 Nov 2014 05:27:22 +0000

Hi,

I meant to copy the mailing list on two emails regarding my (and
Bernard’s) action items.  I mis-spelled rtcweb.  So here the content
integrated into a single email.  (Bernard and Adam: there’s nothing new
herein.)

Hi,
So here my proposal, with respect to H.264 baseline:

MUST:
Filler payload.  You see this occasionally from mostly early generation)
encoder chips.  It¹s basically a way to waste bits so to fulfill buffer
constraints.  Implementation complexity is zero: throw away bits.

Full frame freeze: used in video switching MCUs, to ensure a stable
decoded displayed picture while switching channels in probably a
media-unaware way.  Commonly used in H.32x legacy environments and also in
some SIP MCUs.

SHOULD:
vacat

MAY:
User data T.35 registered, and user data unregistered
Especially the latter you see occasionally for proprietary extensions.
It¹s a good idea to advise implementers that a bitstream may contain that
kind of stuff, even if they don¹t act on it.

Display orientation SEI, as it is referenced in section 4.

With respect to freeze request, below a slightly edited version of an
email that I sent to Adam when he was preparing the pre IETF-91 draft.

Stephan

Freeze-request and freeze-release is something used in the video conf
industry for ages.
Consider a four-way conference with endpoints A, B,  C, and D, and a
video-switching MCU.  A, B, C are all sending pictures to the MCU.  For
receiving endpoint A, the MCU selects either bitstream sent by B or by
endpoint C for forwarding to A.  The selection can be based on voice
activation, conf control, manual request, whatever.  When switching from
B¹s video to C¹s video, the following happens:

1. In-band signaling ³freeze picture request² inserted by the MCU into the
bitstream forwarded to A.  This freezes the display (though not
necessarily the decoding) of the bitstream.
2. MCU sends full intra request to endpoint A, using a signaling protocol.
Meanwhile, it continues to forward all the bitstreams as usual.  Service
to endpoints B, C, and D continue uninterrupted.
3. In some systems supporting dirty refresh points, the MCU starts
forwarding bitstream C to A, resulting in garbage (until the dirty refresh
has cleaned up the picture.  The garbage is luckily not displayed, thanks
to the freeze request.  In other systems, the MCU waits for an intra
picture from A.
4. As soon as the MCU knows that the picture buffer in receiving endpoint
A has been cleaned up (though intra, or dirty refresh), it
sends‹-typically via a signaling protocol like H.245--a freeze-release.
At that point, endpoint A starts displaying again.

The freeze request is in the H.xxx series standards an SEI message.  In
H.261, it is a bit in the the picture header.  The reason is that the
freeze request has to be dealt with synchronously with the bitstream (on a
per picture granularity) so to avoid displaying garbage.  The
freeze-release, OTOH, can be in the signaling, because it does not matter
overly much when the picture is released--faster is better from an user
experience viewpoint, but the freeze only affects the one decoding
endpoint that has decided it wants to see something else, so let this one
suffer the consequences (resulting from delay in the signaling).

This is why pretty much all profiling specs in the video conferencing
industry require freeze request to be supported in the video bitstream,
and it is also why it is available in ITU and MPEG specs since
standardized video compression was developed.

One could argue that pure video switching MCUs are a thing of the past,
and I would mostly concur.  However, in the context of large video
conferences (dozens or hundreds of participants), and also on systems with
limited screen real-estate, video switching continues to be required.
With respect to state-of-the-art technologies, the advent of selective
forwarding units and layered coding also requires freeze requests (in the
enhancement layer) for similar reasons as described above.

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 26 Nov 2014 05:32:16 +0000

Indeed.
Thanks for spotting this.
Stephan
(We could add a whole lot more about transfer characteristics, but perhaps
this doc is not really concerned with those details.)

On 11/25/14, 21:25, "Timothy B. Terriberry" <[email-masked]>
wrote:

>_______________________________________________
>rtcweb mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Daniel-Constantin Mierla <[email-masked]>
Date: Wed, 26 Nov 2014 10:15:56 +0100

IMO, mandatory codec revisiting has to be done for all webrtc entities.
Why not doing it for web browsers when one codec is proved to be royalty
free? Are the specs going to force any browser flavour to pay many forever?

Btw, from the minutes sent on the mailing list, I understood that the
decision is to be confirmed on the list (which is the norm for IETF
anyhow). Unless I missed some messages, I haven't seen any discussion
here following the last IETF meeting, exposing the conclusion of the
meeting and asking to have a decision on that.

Daniel

On 26/11/14 00:33, Adam Roach wrote:
> need your feedback on the remaining open issues (1, 4, and 5, below).
> If you are CCed on this mail, it's because you took an action item in
> Action item for Bernard Aboba and Stephan Wenger: collect list of
> H.264 SEI messages we want to call out as MUST and/or SHOULD.
> ____
>
> Open Issue 1: If you don't want sRGB to be the default color space,
> say something now.
>
> Open Issue 2 (closed): Took "screen source video metadata" issue out
> of the document, as this is an RTP issue, not a codec issue.
>
> Open Issue 3 (closed): New text in "stream orientation" section
> calling out various MUST/SHOULD/MAY levels, based on what I took away
> from the mic in Hawaii. If you care about this, double-check that
> these are what you expect them to be.
>
> Open Issue 4: In Toronto, it was suggested that we don't need to
> specify support for "bilinear" and "none" filter styles, since they're
> already required. I'm happy to remove the statement if someone can
> point me to where this is otherwise mandated. If no one comes forward
> with a citation, I'm going to close this open issue as harmless.
>

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

---

From: Victor Pascual Avila <[email-masked]>
Date: Wed, 26 Nov 2014 15:25:56 +0100

> free? Are the specs going to force any browser flavour to pay many forever?

Agree

> meeting and asking to have a decision on that.

Agree

-Victor

-- 
Victor Pascual Ávila

---

From: Adam Roach <[email-masked]>
Date: Wed, 26 Nov 2014 11:09:01 -0600

On 11/26/14 03:15, Daniel-Constantin Mierla wrote:
> free?

The rationale here is browser accommodation for very constrained "WebRTC 
compatible" devices, which will most typically be concerned with browser 
interoperation. The example that's been tossed around in this space is 
the "WebRTC doorbell" -- when someone rings the bell, you navigate to 
its interface using WebRTC, and stream an image from a small, embedded 
camera. In these kinds of very-low-cost devices, you're going to likely 
see hardware video encoding (e.g. do a web search for NVS2200), which 
will likely be one codec or another, but not both.

The goal is continued browser support for such devices in perpetuity.

> meeting and asking to have a decision on that.

Sure. I'll leave it to the chairs to cross their t's and dot their i's 
how they see fit, but it's pretty clear which direction the wind is 
blowing. Frankly, if I were chair [1], I would consider the on-list 
conversation from November 9th through November 15th (67 messages in 
total) to be sufficient ratification for the proposal, and the in-person 
discussion simply a final punctuation mark on this long-languishing 
chapter.

If the on-list and in-person conversations had been pointed different 
directions from each other, I'd definitely be holding off trying to get 
the draft language nailed down. But they weren't, so I'm not.

/a

_____
[1] I'm not second-guessing what the chairs are doing here, mind you -- 
I'm just pointing out that they appear to be choosing their actions out 
of an abundance of caution rather than absolute necessity.

---

From: Daniel-Constantin Mierla <[email-masked]>
Date: Thu, 27 Nov 2014 11:37:42 +0100

On 26/11/14 18:09, Adam Roach wrote:
> The rationale here is browser accommodation for very constrained
> "WebRTC compatible" devices, which will most typically be concerned
> with browser interoperation. The example that's been tossed around in
> this space is the "WebRTC doorbell" -- when someone rings the bell,
> you navigate to its interface using WebRTC, and stream an image from a
> small, embedded camera. In these kinds of very-low-cost devices,
> you're going to likely see hardware video encoding (e.g. do a web
> search for NVS2200), which will likely be one codec or another, but
> not both.
>

That is the decision of a business entity to go for a technology that
can add extra costs to the user of the product. I couldn't find that as
a guiding rule for IETF versus the rule of preferring technologies with
no IPR.

Similar decisions were taken by ITU/GSMA and cell phone manufacturers
with AMR or other things, which haven't impacted any decision on WebRTC
specs (as it should be).

> total) to be sufficient ratification for the proposal, and the
> in-person discussion simply a final punctuation mark on this
> long-languishing chapter.
>
> directions from each other, I'd definitely be holding off trying to
> get the draft language nailed down. But they weren't, so I'm not.

I didn't get the impression that was any clear conclusion on anything,
including if this is the right proposal (e.g., there were voices arguing
about inability to tailor open source browsers for particular needs,
including form those that as some point welcomed the proposal) as well
as what and how should be mandatory for what so ever form (terminology)
of webrtc endpoints.

Besides the well known group of interests jumping on stage with the
'yes', there were real concerns put on table that seems to be ignored.

Moreover, your proposal with considering a past discussions for
something that was not made clear as a decision making process to a
sensitive subject, looks like changing the rules of the game after the
match. Then, using the in-person meeting of a bunch of people in a
far-end corner of the world (for most of the population) to end this
subject, again, without a prior notification, definitely is not a
transparent process, but more an obscure decision system, which I though
it is not how IETF aims to work.

Daniel

>
> /a
>
> _____
> [1] I'm not second-guessing what the chairs are doing here, mind you
> -- I'm just pointing out that they appear to be choosing their actions
> out of an abundance of caution rather than absolute necessity.