Subject: [payload] Submission of new versions of H.265/HEVC payload format

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 11 Jun 2013 17:00:19 +0000

Dear All,

We have just submitted versions 02 and 03 of draft-schierl-payload-rtp-h265, for which the links are as follows:

Version 02:
URL:             http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-02.txt
Htmlized:        http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-02
Diff:            http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-02

Version 03:
URL:             http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-03.txt
Htmlized:        http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-03
Diff:            http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-03

Version 02 includes huge changes compared to the earlier submitted version 01. In a short summary, the authors have worked hard to try to make the design complete, from both the payload structure and the signaling points of view. Compared to version 02, version 03 only contains a correction of the email address of an author.

Due to that the industry is eager to deploy H.265/HEVC and other standards bodies such as 3GPP rely on the payload format for support of H.265/HEVC in RTP based video services, we need to progress this draft relatively quickly. Therefore, we would like to request quick reviews from interested parties and also request for the WG status of this draft. 

BR, YK (on behalf of the authors)

---

From: Rickard Sjöberg <[email-masked]>
Date: Thu, 20 Jun 2013 16:16:46 +0000

Dear all,

I have taken a first look at draft-schierl-payload-rtp-h265-03 and have some questions/comments.

My first question is what extensions of HEVC that draft-schierl-payload-rtp-h265 is intended to cover? The draft mentions "future  scalable  or  3D  video  coding  extensions  of  this  specification", what extensions do you refer to? Is the intent to cover the all HEVC extensions in a single payload specification?

Another question for my understanding is why "Receivers MUST pass picture timing SEI messages and decoding unit information SEI messages to the decoder"?

In "5. Packetization Rules" in Section 4.7, there are 5 rules. The fourth one does not seem to be a rule since there is no "MUST" or "SHOULD" in the corresponding text. Further, the fifth one discourages using FUs, what are the reasons behind that? 

In 8.2 and 8.3, the feedback messages are using PicOrderCntVal. Isn't there a possiblity that the MSB of PicOrderCntVal may differ between the encoder and decoder?

The use of spatial-segmentation-idc may need some updates to be useful, let me come back to that later on.

I have also a number of spelling suggestions for your considerations:

Section 1.1:
Replace "community" by "communities"

Section 1.1.1:
Replace "In addition, interpolation filter" by "In addition, the interpolation filter"
Replace "skipping the transform coding" by "skipping the transform" (transform_skip_flag of HEVC skips the transform, not the coding)

Section 1.1.2:
Replace:
"2) An indication of whether there is any parameter set within the current coded video sequence that updates another parameter set of the same type preceding in decoding order."
by:
"2) An indication of whether there is no parameter set update in the coded video sequence."
since that is what no_parameter_set_update_flag in HEVC indicates.

Section 4.7:
Replace "Type" in Figure 10, and "Type:  6 bits" by "FU Type" and "FU Type:  6 bits" to avoid confusion with the Type in the payload header.

Section 6:
Replace "recovery" by "recover"

Section 7.1:
Replace "level-id" by "level-idc"
Replace "sub_layer_leve_idc[j]" by "sub_layer_level_idc[j]"

Section 8:
Replace "two  payload-specific  feedback  messages" by "a  payload-specific  feedback  message", since only SPLI is "Assigned in this memo".
Also, consider removing references to H.264 in the last paragraph of Section 8 since the memo is about HEVC.

Section 8.1:
Seems to me that "There MUST be exactly one RPSI contained in the FCI field" should be "There MUST be exactly one SPLI contained in the FCI field"

Best Regards,
Rickard Sjöberg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 11 juni 2013 19:00
To: [email-masked]
Subject: [payload] Submission of new versions of H.265/HEVC payload format

Dear All,

Version 02:

Version 03:

BR, YK (on behalf of the authors)
_______________________________________________
payload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/payload

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Sun, 23 Jun 2013 23:00:16 +0000

Hi Rickard,

Thanks for the careful review and the comments. Please see inline below.

BR, YK

> From: Rickard Sjöberg [mailto:[email-masked]]
> Sent: Thursday, June 20, 2013 9:17 AM
> To: Wang, Ye-Kui; [email-masked]
> Subject: RE: Submission of new versions of H.265/HEVC payload format
> 
> Dear all,
> 
> My first question is what extensions of HEVC that draft-schierl-payload-rtp-
> h265 is intended to cover? The draft mentions "future  scalable  or  3D  video
> coding  extensions  of  this  specification", what extensions do you refer to?
> Is the intent to cover the all HEVC extensions in a single payload specification?
> 

[YK] That is in the semantics of LayerId (nuh_layer_id), where "this specification" should be replaced with "[HEVC]". It is sort of copy-and-paste error. Thanks for the good catching. No, there is no intention to cover HEVC extensions in this payload specification, though we intentionally to have the depacketization process and the use with feedback messages to work for HEVC scalability and 3DV extensions.

> Another question for my understanding is why "Receivers MUST pass picture
> timing SEI messages and decoding unit information SEI messages to the
> decoder"?

[YK] Generally, RTP receivers should/must pass all NAL units specified by the video coding spec to the video decoders. This is emphasized here for the two timing related SEI messages after it is said that picture output timing in RTP timestamps should be used instead (to avoid giving a wrong impression to readers that the SEI messages should be ignored). If an RTP receiver discards the SEI messages, then HRD conformance checking for the bitstream that was possible would be disabled, and also the information related to frame doubling or tripping would be lost.

> 
> In "5. Packetization Rules" in Section 4.7, there are 5 rules. The fourth one
> does not seem to be a rule since there is no "MUST" or "SHOULD" in the
> corresponding text.

[YK] I see your point, it is only a "MAY" rule. An alternative to put this as a packetization rule is to put it as part of the semantics of NAL unit header (1.1.4), but that sub-section belongs to an overview wherein normative language (even "MAY") should not appear. One solution to this is to insert "Payload Header Usage" after 4.1 "RTP Header Usage" and include at least this item there. I will do this in the next version unless there is a better suggestion.

> Further, the fifth one discourages using FUs, what are
> the reasons behind that?

[YK] The bullet item only discourages using FUs for live-encoding scenarios, wherein dependent slice segments should be used instead. This was added after a discussion (among the authors) on the need of whether to specify a payload structure for mixed FU and aggregation packets and from that discussion we concluded that encoding dependent slice segments at source coding level already allows for the use cases and using of FUs for live-encoding does not make sense. Please let us know if you think differently.

> In 8.2 and 8.3, the feedback messages are using PicOrderCntVal. Isn't there a
> possiblity that the MSB of PicOrderCntVal may differ between the encoder
> and decoder?

[YK] Good question. In theory this is indeed possible when random accessing to a bitstream is performed. However, it would not happen in conversational applications where the feedback messages may be used. Therefore, to me this would just work just fine. But please feel free to make other suggestions.

> me come back to that later on.

[YK] OK. Look forward to seeing your input herein.

> Replace "community" by "communities"

[YK] I guess both are OK. But anyway I just changed it per your suggestion (for the next version).

> Replace "In addition, interpolation filter" by "In addition, the interpolation

[YK] Thanks - done.

> Replace "skipping the transform coding" by "skipping the transform"
> (transform_skip_flag of HEVC skips the transform, not the coding)
> 

 [YK] Done.

> Section 1.1.2:
> Replace:
> "2) An indication of whether there is any parameter set within the current
> coded video sequence that updates another parameter set of the same type
> preceding in decoding order."
> by:

[YK] Changed the wording to "An indication of whether there is no parameter set within the current coded video sequence that updates another parameter set of the same type preceding in decoding order."

> Section 4.7:
> Replace "Type" in Figure 10, and "Type:  6 bits" by "FU Type" and "FU Type:  6
> bits" to avoid confusion with the Type in the payload header.
> 

[YK] Good point again. I will change the field name to be "FuType".

> Replace "recovery" by "recover"

[YK] Done.

> Replace "level-id" by "level-idc"

[YK] We are consistently using "id" for profile-id, level-id, max-recv-level-id etc, similarly as in RFC 6184, instead of "idc".

> Replace "sub_layer_leve_idc[j]" by "sub_layer_level_idc[j]"

 [YK] Done.

> 
> Section 8:
> Replace "two  payload-specific  feedback  messages" by "a  payload-specific
> feedback  message", since only SPLI is "Assigned in this memo".

[YK] Good catch again. Changed.

> 
> Section 8.1:
> Seems to me that "There MUST be exactly one RPSI contained in the FCI
> field" should be "There MUST be exactly one SPLI contained in the FCI field"

[YK] Another good catch. Changed.

> Version 02 includes huge changes compared to the earlier submitted version
> 01. In a short summary, the authors have worked hard to try to make the
> design complete, from both the payload structure and the signaling points of
> view. Compared to version 02, version 03 only contains a correction of the
> email address of an author.
> 
> Due to that the industry is eager to deploy H.265/HEVC and other standards
> bodies such as 3GPP rely on the payload format for support of H.265/HEVC in
> RTP based video services, we need to progress this draft relatively quickly.
> Therefore, we would like to request quick reviews from interested parties
> and also request for the WG status of this draft.
>

---

From: Tom Kristensen <[email-masked]>
Date: Thu, 27 Jun 2013 15:33:37 +0200

We have done a first review of the new version of this draft. Seems
promising. However, we are missing a couple of things.

First, we would like to include a max-fps parameter in line with the
proposed draft draft-kristensen-payload-rtp-h241param-00. A similar
parameter has been part of H.241 for while and will eventually be valid for
H.265 in H.323 land too. Note that this is a limiting parameter, different
from the other max-* parameters (max-ls, max-lps, max-cpb, max-dpb and
max-br).

We propose max-fps as something along the lines of:
   ------
   max-fps:  The value of max-fps is an integer indicating the maximum
      frame rate in units of hundredths of frames per second that can be
      efficiently received.  The signaled highest level is conveyed in
      the value of the profile-level-id parameter or the max-recv-level
      parameter.  The max-fps parameter MAY be used to signal that the
      receiver has a constraint in that it is not capable of decoding
      video efficiently at the full frame rate that is implied by the
      signaled highest level and, if present, one or more of the
      parameters max-ls, max-lps or max-br.

      Notice that the value of max-fps is not necessarily the frame rate
      at which the maximum frame size can be sent, it constitutes a
      constraint on maximum frame rate for all resolutions.

         Informational note: The max-fps parameter is semantically
         different from max-ls, max-lps, max-cpb, max-dpb,
         and max-br in that max-fps is used to signal a constraint,
         lowering the maximum frame rate from what is implied by the
         signaled MaxLumaSR and MaxLumaPS.

      The encoder MUST use a frame rate equal to or less than this
      value.  In cases where the max-fps parameter is absent the
      encoder is free to choose any frame rate according
      to the highest signaled level and any signaled optional
      parameters.
   ------

Second, we are missing an ability to signal support for tiles (number of
tiles used/to expect) and maximum tile size. The ability to signal support
for a number of tiles higher than what is implied for the signalled level
(as listed in Table A-1 in the HEVC specification. These two parameters
have the same semantics as the existing max-* parameters (max-ls, max-lps,
max-cpb, max-dpb and max-br).

We propose max-tr (or max-tile-rows) and the equivalent description for
max-tc (or max-tile-cols) along the lines of:
   ------
   (Will be introduced toghether with max-ls, max-lps, max-cpb, max-dpb,
max-br, as parameters that "MAY be used to signal the capabilities of a
receiver implementation.")
capabilities...:

   max-tr:
      The value of max-tr is an integer indication the maximum number
      of tile rows. The max-tr parameter signals that the receiver is
      capable of decoding video with a larger number of tile rows than
      is required by the signaled highest level.

      When max-tr is signaled, the receiver must be able to decode NAL
      unit streams that conform to the signaled highest level, with the
      exception that the MaxTileRows value in Table A-1 of [HEVC] for
      the signaled highest level is replaced with the value of max-tr.
      The value of max-tr MUST be greater than or equal to the value of
      MaxTileRows given in Table A-1 of [HEVC] for the highest
      level. Senders MAY use this knowledge to send pictures utilizing
      a larger number of tile rows than is indicated in the signaled
      highest level.

   max-tc:
      (As for max-tr with s/max-tr/max-tc/
                          s/MaxTileRows/MaxTileCols/
                          s/rows/colums)
   ------

The last tile related parameter is a new parameter named max-lts to specify
maximum luma tile size to make sure the decoder doesn't receive a stream
with too large resolution and/or bitrate in one single tile. The max-lts
parameter is a limiting parameter - as described for max-fps above.

We propose max-lts to signal the maximum size allowed in one tile along the
lines of:
   ------
   max-lts:
      The value of max-lts, Maximum Luma Tile Size, is an integer
      indicating the maximum tile size in units of luma samples. The
      max-lts parameter signals the maximum size of one tile that the
      decoder is able to decode.

         Informational note: The max-fps parameter is semantically
         different from max-ls, max-lps, max-cpb, max-dpb,
         and max-br in that max-lts is used to signal a constraint
         on the maximum tile size.

      The encoder MUST use a tile size equal to or less than this
      value.  In cases where the max-lts parameter is absent the
      encoder is free to choose any tile size according
      to the highest signaled level and any signaled optional
      parameters.
   ------

Please consider these additions. More text needed to merge the new
parameters into the media subtype specification, offer/answer rules and so
on - of course. To be dealt with later.

-- Tom

On 11 June 2013 19:00, Wang, Ye-Kui <[email-masked]> wrote:

> Dear All,
>
> We have just submitted versions 02 and 03 of
> draft-schierl-payload-rtp-h265, for which the links are as follows:
>
> Version 02:
> URL:
> http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-02.txt
> Htmlized:
> http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-02
> Diff:
> http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-02
>
> Version 03:
> URL:
> http://www.ietf.org/internet-drafts/draft-schierl-payload-rtp-h265-03.txt
> Htmlized:
> http://tools.ietf.org/html/draft-schierl-payload-rtp-h265-03
> Diff:
> http://www.ietf.org/rfcdiff?url2=draft-schierl-payload-rtp-h265-03
>
> design complete, from both the payload structure and the signaling points
> of view. Compared to version 02, version 03 only contains a correction of
> the email address of an author.
>
> RTP based video services, we need to progress this draft relatively
> quickly. Therefore, we would like to request quick reviews from interested
> parties and also request for the WG status of this draft.
>

-- 
# Cisco                         |  http://www.cisco.com/telepresence/
## [email-masked]  |  http://www.tandberg.com
###                               |  http://folk.uio.no/tomkri/

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 27 Jun 2013 17:28:18 +0000

Hi Tom,

Thanks for the review and the suggestions. I have some comments on the suggestions.

On max-fps: The following paragraph in draft-kristensen-payload-rtp-h241param-00 seems to be the main reason for the introduction of max-fps. Just for a better understanding, could you please clarify why the receiver side would have a preferred frame rate? Does that relate to rendering/display capability? If that is the case, then discarding of the additional frames should be done by the rendering module instead of the decoder, as conforming decoders should just output frames that are indicated, by the bitstream, to be output.

   If the encoder chooses to send at a higher frame rate than preferred
   by the receiver side, the decoder will normally discard the
   additional frames after decoding them.  The transmission of the extra
   frames and the processing of frames to just discard them are wasteful
   and the bandwidth and processing could be used more effectively.

Not relevant for H.265/HEVC payload, but for H.264/AVC payload in RFC 6184, would introduction of such a parameter cause any IOP issue between a legacy sender and a new receiver? I think that should be analyzed in draft-kristensen-payload-rtp-h241param-00. Certainly, it would also be good to clarify the above question related to rendering/display capability in draft-kristensen-payload-rtp-h241param-00 too.

On max-lts: It makes sense to me, but we would need to specify a lower bound of the value, such that it would at least not contradict with the min tile size implied by other parameters.

On max-tr and max-tc: It seems that they are saying that there are scenarios where you would need more tiles than allowed by the indicated level. Could you please give an (practical) example that this is needed? I am asking this because in JCT-VC there had been a lot of discussions after which the level limits of MaxTileRows and MaxTileCols were concluded.

BR, YK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Kristensen
Sent: Thursday, June 27, 2013 6:34 AM
To: Wang, Ye-Kui
Cc: Geir Arne Sandbakken; [email-masked]; Tom Kristensen
Subject: Re: [payload] Submission of new versions of H.265/HEVC payload format

We have done a first review of the new version of this draft. Seems promising. However, we are missing a couple of things.

First, we would like to include a max-fps parameter in line with the proposed draft draft-kristensen-payload-rtp-h241param-00. A similar parameter has been part of H.241 for while and will eventually be valid for H.265 in H.323 land too. Note that this is a limiting parameter, different from the other max-* parameters (max-ls, max-lps, max-cpb, max-dpb and max-br).

Second, we are missing an ability to signal support for tiles (number of tiles used/to expect) and maximum tile size. The ability to signal support for a number of tiles higher than what is implied for the signalled level (as listed in Table A-1 in the HEVC specification. These two parameters have the same semantics as the existing max-* parameters (max-ls, max-lps, max-cpb, max-dpb and max-br).

We propose max-tr (or max-tile-rows) and the equivalent description for max-tc (or max-tile-cols) along the lines of:
   ------
   (Will be introduced toghether with max-ls, max-lps, max-cpb, max-dpb, max-br, as parameters that "MAY be used to signal the capabilities of a receiver implementation.")
capabilities...:

   max-tr:

   max-tc:

The last tile related parameter is a new parameter named max-lts to specify maximum luma tile size to make sure the decoder doesn't receive a stream with too large resolution and/or bitrate in one single tile. The max-lts parameter is a limiting parameter - as described for max-fps above.

We propose max-lts to signal the maximum size allowed in one tile along the lines of:
   ------
   max-lts:

Please consider these additions. More text needed to merge the new parameters into the media subtype specification, offer/answer rules and so on - of course. To be dealt with later.

-- Tom

On 11 June 2013 19:00, Wang, Ye-Kui <[email-masked]<mailto:[email-masked]>> wrote:
Dear All,

Version 02:

Version 03:

BR, YK (on behalf of the authors)
_______________________________________________
payload mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/payload

--
# Cisco                         |  http://www.cisco.com/telepresence/
## [email-masked]<mailto:[email-masked]>  |  http://www.tandberg.com
###                               |  http://folk.uio.no/tomkri/