Subject: Re: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 11 Apr 2014 16:26:35 +0000

Good. It seems that we are settled down here. I will implement the agreed changes in the next version, which I hope can be submitted soon (next week or the week after).

BR, YK

-----Original Message-----
From: Jonatan Samuelsson [mailto:[email-masked]] 
Sent: Wednesday, March 12, 2014 5:18 AM
To: Wang, Ye-Kui; Jonathan Lennox
Cc: payload issue tracker; [email-masked]; Stephan Wenger; [email-masked]
Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC

Hi, 

Regarding Jonathan's question:
"Bare SHOULDs (i.e., a statement that says something SHOULD be done, without explaining the circumstances in which an implementation would not do it) are generally considered undesirable in IETF standards these days.  (This is a stylistic change since RFC 3551 was written.)  What do you consider the circumstances in which it would be appropriate for an H.265 RTP implementation not to set the marker bit?"

In fact I would prefer if it was explicitly stated that the marker bit MUST be set on the last packet in an RTP stream of an access unit and that it MUST NOT be set on a packet that is not the last packet of an RTP stream of an access unit. 

One concrete example of where a "lazy" MANE would be tempted not to set marker bit correctly is when an RTP stream containing two temporal layers is split up to two different RTP streams based on temporal_id. In order two deduce which packet to set the marker bit on in each stream the MANE would have to buffer the entire access unit before being able to forward the last packet of the stream corresponding to the higher temporal_id.

Another example was the one I mentioned earlier. A "lazy" MANE that (for some reason) removes suffix SEI messages (perhaps it has somehow been informed that the receiver is not capable of making use of them) without buffering preceding packets will not set the marker bit on the last packet if it has already been forwarded. 

A third example is for SHVC or MV-HEVC in which a MANE extracts a base layer from a single RTP stream containing multiple layers. If that stream is supposed to be correct according to the HEVC payload format the MANE will most probably have to buffer and wait for all layers of that access unit before deducing which packet was the last packet of the base layer.

Anyway. I appreciate the constructive discussion on the topic (thanks) and I think the suggestion on the table:
 
" Marker bit (M): 1 bit
Set for the last packet, carried in the current RTP stream, of an access unit, in line with the normal use of the M bit in video formats, to allow an efficient playout buffer handling." 

is ok and an improvement compared to the original text in the draft but I would appreciate if it was made clear that this is to be interpreted as a MUST (not a SHOULD) and that some more text is added to, as Jonathan suggested:

"explicitly pointing out that in MST mode, if an access unit appears in multiple packet streams, the marker bit is set on each packet stream's last packet of the access unit."

Best Regards Jonatan

-----Original Message-----
From: Wang, Ye-Kui [mailto:[email-masked]] 
Sent: den 10 mars 2014 17:30
To: Jonathan Lennox; Jonatan Samuelsson
Cc: payload issue tracker; [email-masked]; Stephan Wenger; [email-masked]
Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC

I have basically the same opinion here as Jonathan.

Just one aspect regarding Jonatan's following comment:

> Just because you have received all packets with the same timestamp it is not necessarily the case that you can safely display the picture in that access unit, e.g. if you in the next access unit receive a picture with lower timestamp. Thus, in my opinion, the most useful information that you can get from the marker bit is whether you have received all VCL NAL units of a picture or not.

[YK] I think your first sentence is correct, but I don't see the logic that this true statement can lead to the conclusion that "the most useful information that you can get from the marker bit is whether you have received all VCL NAL units of a picture or not". To me, if you have reordering in encoding, then you are not doing low-delay, and then the marker bit is less useful than in low-delay where there is no reordering in encoding.

BR, YK

-----Original Message-----
From: Jonathan Lennox [mailto:[email-masked]] 
Sent: Monday, March 10, 2014 9:13 AM
To: Jonatan Samuelsson
Cc: Wang, Ye-Kui; payload issue tracker; [email-masked]; Stephan Wenger; [email-masked]
Subject: Re: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC

On Mar 10, 2014, at 9:17 AM, Jonatan Samuelsson <[email-masked]> wrote:

> Thanks for your response Ye-Kui (sorry for my late reply),
> 
> In my opinion it does make sense to distinguish between VCL data and non-VCL data. Just because you have received all packets with the same timestamp it is not necessarily the case that you can safely display the picture in that access unit, e.g. if you in the next access unit receive a picture with lower timestamp. Thus, in my opinion, the most useful information that you can get from the marker bit is whether you have received all VCL NAL units of a picture or not.

I disagree.  I don't think VCL and non-VCL NAL units should be distinguished for the marker bit.  Presumably any encoder which is sending suffix NAL units (and I would expect them to be rare) is doing so because it believes they'll be useful for the decoders to fully decode the access unit.

If a later packet contains an earlier timestamp, you're either doing frame reordering (in which case it *is* safe to decode the earlier frame in decode order) or else using DON (in which case you need to track DON values rather than sequence numbers).

The marker bit - by definition - can't carry more than one bit of data, so there's no way to tell just from that bit whether it's safe to decode a frame given arbitrary earlier packet loss.  Without packet loss, it should always be possible to decode an access unit when the marker bit is received, if all earlier access units in decode order (since the last refresh point) have already been decoded.

> Anyway, let me ask one question on the solution that you were suggesting. 
> 
> " Marker bit (M): 1 bit
> Set for the last packet, carried in the current RTP stream, of an access unit, in line with the normal use of the M bit in video formats, to allow an efficient playout buffer handling."
> 
> Would it be equivalent to:
> 
> " Marker bit (M): 1 bit
> Set when the next packet (if any), in the same RTP stream, has a different timestamp than the current packet."

They are equivalent when not using DON (or more specifically when sprop-depack-buf-nalus > 0).  When you are using DON, they are different; and I think the original definition is more useful.

> Another question related to
> 
>   Most of these video encodings also specify that the marker bit of the
>   RTP header SHOULD be set to one in the last packet of a video frame
>   and otherwise set to zero.  Thus, it is not necessary to wait for a
>   following packet with a different timestamp to detect that a new
>   frame should be displayed.
> 
> It uses "SHOULD" from RFC 2119 but in the H.265 payload format "SHOULD" is not used. Is the intended interpretation:
> 
> " Marker bit (M): 1 bit
> SHOULD be set for the last packet..."?

Bare SHOULDs (i.e., a statement that says something SHOULD be done, without explaining the circumstances in which an implementation would not do it) are generally considered undesirable in IETF standards these days.  (This is a stylistic change since RFC 3551 was written.)  What do you consider the circumstances in which it would be appropriate for an H.265 RTP implementation not to set the marker bit?

> Best Regards Jonatan
> 
> -----Original Message-----
> From: Wang, Ye-Kui [mailto:[email-masked]]
> Sent: den 6 mars 2014 22:52
> To: Jonatan Samuelsson; Jonathan Lennox
> Cc: payload issue tracker; [email-masked]; 
> Stephan Wenger; [email-masked]
> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
> 
> [Jonatan]: I don't think this text is really up to date with HEVC in the sense that the actual video frame (which in my opinion, for the HEVC case, is the VCL NAL units of an access unit) can be completely received even though there are more packet(s) that are following with the same time stamp.
> 
> I don't understand the above sentence. I think the receiver can only conclude that an access unit is completely received only when no more packets with the same timestamp is coming soon.
> 
> For other parts of your new message below: Any data, including non-VCL data, associated with a picture in the bitstream is supposed to be used by the decoder side (not just the decoder) to do something that can be helpful in processing (decoding including error concealment and so on, display, etc.) of the decoded picture, even not needed for NORMATIVE decoding of the picture. Thus, only when the decoder side receives all data associated with a picture, can it be sure that no more useful information may come before sent the decoded picture to the display. Therefore, we should not change the wording from "the last NAL unit" to "the last VCL NAL unit". Also, without making the setting to be stream specific, as I suggested and Jonathan agreed, multi-layer extensions will be problematic.
> 
> BR, YK
> 
> 
> -----Original Message-----
> From: Jonatan Samuelsson [mailto:[email-masked]]
> Sent: Thursday, March 06, 2014 2:59 AM
> To: Wang, Ye-Kui; Jonathan Lennox
> Cc: payload issue tracker; [email-masked]; 
> Stephan Wenger; [email-masked]
> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
> 
> Thanks Ye-Kui and Jonathan for your responses.
> 
> Regarding Section 5 of RFC 3551:
> 
>   Most of these video encodings also specify that the marker bit of the
>   RTP header SHOULD be set to one in the last packet of a video frame
>   and otherwise set to zero.  Thus, it is not necessary to wait for a
>   following packet with a different timestamp to detect that a new
>   frame should be displayed.
> 
> I don't think this text is really up to date with HEVC in the sense that the actual video frame (which in my opinion, for the HEVC case, is the VCL NAL units of an access unit) can be completely received even though there are more packet(s) that are following with the same time stamp.
> 
> Since none of the suffix SEI messages has any normative effect on the output of the decoder, why should it be necessary to wait for example for a decoded_picture_hash SEI before detecting that a frame can be displayed?
> 
> Ye-Kui, you earlier wrote:
> "Anyway, I admitted that the current semantics are not fully future-proof regarding suffix SEI messages. In looking into future multi-layer extensions, it appears that simply changing the wording from "last NAL unit" to "last VCL NAL unit" is not good enough."
> 
> But if (as I initially suggested) the change is from "last NAL unit of an access unit" to "last VCL NAL unit of a picture" I guess it could work also for future extensions in which there will be several pictures in the same access unit. Or do you see a problem here?
> 
> Best Regards Jonatan
> 
> -----Original Message-----
> From: Wang, Ye-Kui [mailto:[email-masked]]
> Sent: den 6 mars 2014 01:24
> To: Jonathan Lennox
> Cc: Jonatan Samuelsson; payload issue tracker; 
> [email-masked]; Stephan Wenger; 
> [email-masked]
> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
> 
> Agree to make the intention clearer, e.g. by adding a note.
> 
> On your last question: I think you are right, a non-VCL suffix NAL unit, e.g. an SEI, end of sequence, or end of bitstream NAL unit, could have nuh_layer_id equal to 0, and thus in MST the last NAL unit of an access unit may actually be carried in the last packet from the "base" RTP stream. And yes this actually does not matter for using of the marker bit for the purpose described below. 
> 
> BR, YK
> 
> -----Original Message-----
> From: Jonathan Lennox [mailto:[email-masked]]
> Sent: Wednesday, March 05, 2014 4:05 PM
> To: Wang, Ye-Kui
> Cc: Jonatan Samuelsson; payload issue tracker; 
> [email-masked]; Stephan Wenger; 
> [email-masked]
> Subject: Re: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
> 
> This seems correct to me.  I would suggest, however, explicitly pointing out that in MST mode, if an access unit appears in multiple packet streams, the marker bit is set on each packet stream's last packet of the access unit.
> 
> I think this definition works even when DONs are in use (when sprop-depack-buf-nalus is > 0), since an AP can only contain packets from a single access unit.  If packets from different access units are interleaved (in RTP order) using DON, the packet stream will look odd to a naive RTP analysis, but I believe the marker bit will still be useful, allowing low-delay decoders to flush complete access units out of the de-packetization buffer early.
> 
> Ye-Kui, a question about your comment, though: are we sure that the final packet of the access unit will in fact always be the final packet of the highest layer?  In particular, in SHVC or MV-HEVC, presumably we'll be able to split MST streams by nuh_layer_id.  How will the nuh_layer_id be set for trailing SEIs in such streams?  If it's set to 0, then the final packet of the access unit will in fact be sent in a lower layer.  I don't think this actually matters, especially since MST streams will use DON, but we should be careful about our logic.
> 
> 
> 
> On Mar 5, 2014, at 7:20 PM, Wang, Ye-Kui <[email-masked]> wrote:
> 
>> I did a bit of search on the usage of the marker bit for video, and the most relevant description I found was the following paragraph in Section 5 of RFC 3551:
>> 
>>  Most of these video encodings also specify that the marker bit of 
>> the  RTP header SHOULD be set to one in the last packet of a video 
>> frame  and otherwise set to zero.  Thus, it is not necessary to wait 
>> for a  following packet with a different timestamp to detect that a 
>> new  frame should be displayed.
>> 
>> So the key usage of the marker bit is that in some low-delay applications, output/display of a picture can be done before receiving an RTP packet with a different/greater timestamp. Since non-VCL NAL unit should have the same timestamp as the associated picture, thus I think we should not use the wording of "last VCL NAL unit" in determining the setting of the marker bit.
>> 
>> To make the semantics extensible or future-proof for multi-layer extensions, I think we should change it to be RTP stream (or equivalently packet stream) specific, and each receiver can just look at the marker bits of the packets from the highest RTP stream for the usage. And this will also resolve the issue of requiring some entities to change the marker bit in sub-layer scalability usage as Jonaton raised. 
>> 
>> In other words, the normative semantics are to be changed as follows:
>> 
>> Marker bit (M): 1 bit
>> Set for the last packet, carried in the current RTP stream, of an access unit, in line with the normal use of the M bit in video formats, to allow an efficient playout buffer handling.
>> 
>> Please let me know if there is any concern with such a change (and note that we would align the terminology of RTP stream with other drafts, e.g. the Grouping Taxonomy draft, as discussed in responding to Magnus' reviewing comments).
>> 
>> BR, YK
>> 
>> -----Original Message-----
>> From: payload [mailto:[email-masked]] On Behalf Of Wang, 
>> Ye-Kui
>> Sent: Wednesday, March 05, 2014 10:29 AM
>> To: Jonatan Samuelsson; payload issue tracker; 
>> [email-masked]; [email-masked]
>> Cc: [email-masked]
>> Subject: Re: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
>> 
>> Something should be done to the HEVC spec to make it clearer for the decoded picture hash SEI message, like that it shall not be nested and what the values of LayerId and LId should be, or allow add to the list of nestable SEI message. In any case, it does not make sense to me to put a greater value of TId than its associated picture. On user_data_registered_itu_t_t35 and user_data_unregistered suffix SEI messages, only if it is defined by some application then it may make sense to put a greater value of TId than its associated picture - currently I have not seen any such use being defined anywhere.
>> 
>> Anyway, I admitted that the current semantics are not fully future-proof regarding suffix SEI messages. In looking into future multi-layer extensions, it appears that simply changing the wording from "last NAL unit" to "last VCL NAL unit" is not good enough. We might need to specify the semantics of the marker bit to be RTP stream specific. To make sure whether such a change is OK, we also need to check against the usage of the marker bit.
>> 
>> BR, YK
>> 
>> -----Original Message-----
>> From: Jonatan Samuelsson [mailto:[email-masked]]
>> Sent: Wednesday, March 05, 2014 1:07 AM
>> To: Wang, Ye-Kui; payload issue tracker; 
>> [email-masked]; [email-masked]
>> Cc: [email-masked]
>> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
>> 
>> Thanks for your response Ye-Kui,
>> 
>> As far as I understand, out of the 6 SEI messages that have so far been defined in the HEVC spec as possible to use as suffixes (filler_payload, user_data_registered_itu_t_t35, user_data_unregistered, progressive_refinement_segment_end, post_filter_hint, and decoded_picture_hash) there are 3 that are not required to have the same TId as the VCL NAL units of the access unit:  user_data_registered_itu_t_t35, user_data_unregistered and decoded_picture_hash since their payloadType numbers (4, 5 and 132) are not included in the list:
>> 
>> "When a non-nested SEI message has payloadType equal to 2, 3, 6, 9, 15, 16, 17, 19, 22, 23, 45, 47, 128, 131, or 134 (i.e., one of the SEI messages that have payloadType not equal to 0, 1, or 130, and that are allowed to be nested SEI messages), the SEI NAL unit containing the non-nested SEI message shall have TemporalId equal to the TemporalId of the access unit containing the SEI NAL unit."
>> 
>> Maybe it's a mistake in the HEVC spec that 132 is not in this list, but I think the problem would exist even if was. An encoder that (for whatever reason) puts in user_data_registered_itu_t_t35 or user_data_unregistered suffix SEI messages with higher TId would put a MANE in a difficult situation. 
>> 
>> I also think we should consider future extensions of HEVC (i.e. SHVC and MV-HEVC) in which an access unit may contain many pictures with different values of nuh_layer_id. If the marker bit is set for the last packet of an access unit, then once again all the marker bits will end up in the highest layer (stream) and a MANE would have to change the value of marker bits as soon as not all layers (streams) are forwarded.
>> 
>> Best Regards Jonatan
>> 
>> -----Original Message-----
>> From: Wang, Ye-Kui [mailto:[email-masked]]
>> Sent: den 5 mars 2014 01:20
>> To: Jonatan Samuelsson; payload issue tracker; 
>> [email-masked]; [email-masked]
>> Cc: [email-masked]
>> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
>> 
>> Hmm, a decoded picture hash SEI message applies to (and is associated with) the preceding decoding picture in decoding order and thus the TId of the containing suffix SEI NAL unit should be set equal to the TId of the associated picture. Thus in the example, for the suffix SEI NAL unit containing the decoded picture hash SEI message for the first encoded picture should be assigned TId equal to 0. 
>> 
>> According to the HEVC spec, currently the following SEI messages are or can be suffix SEI messages: filler_payload, user_data_registered_itu_t_t35, user_data_unregistered, progressive_refinement_segment_end, post_filter_hint, and decoded_picture_hash. Among these, I think all of the "specified" SEI messages (i.e. the above list excluding user_data_registered_itu_t_t35, user_data_unregistered) contained in a suffix SEI NAL unit would apply to the preceding decoded picture in decoder order and should have the same TId as the associated picture. The only SEI messages that can have a greater TId value than the containing access unit are the three SEI message that can carry HRD information, i.e., buffering_period, pic_timing, and decoding_unit_info, but these can only be prefix SEI messages.
>> 
>> Of course, in theory, future versions of HEVC or some extensions (even defined by an application spec) could define suffix SEI messages that can have a greater TId value than the containing access unit. In that case, what you described could be meaningful, and indeed if a receiver receives the "base sub-layer" stream, some entity would have to set the marker bit of those packets containing the last NAL unit of an access unit for that stream, according to the current semantics.
>> 
>> However, if we change the wording to the last VCL NAL unit as you suggested, and in case a suffix SEI NAL unit of picture A is not carried the packet containing the last VCL NAL unit if the access unit, would the usage of the marker bit be affected, e.g. in playout buffer handling? If the answer is no, then there should be no problem to do the change.
>> 
>> Basically the situation is as follows. There is no concrete problem now. However, there can be a potential problem in the future if a suffix SEI message is defined and can have a greater TId value than the containing access unit. If the suggested change does not affect the usage of the bit, we should do the change now to make the design future-proof. So the key question is whether the suggest change would affect the usage of the bit. 
>> 
>> Does anyone know how the marker bit is used in playout buffering handling? An explanation or a pointer would be greatly appreciated.
>> 
>> BR, YK
>> 
>> -----Original Message-----
>> From: Jonatan Samuelsson [mailto:[email-masked]]
>> Sent: Tuesday, March 04, 2014 1:28 PM
>> To: Wang, Ye-Kui; payload issue tracker; 
>> [email-masked]; [email-masked]
>> Cc: [email-masked]
>> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
>> 
>> Dear Ye-Kui,
>> 
>> Let me try to make the example more specific (and then also see if I have understood the multi-stream transmission concept correctly).
>> 
>> Assume that the sender puts all NAL units with TId 0 in one RTP stream (stream A) and all NAL units with TId 1 in another RTP stream (stream B). Now for the first encoded picture the VCL NAL units are assigned TId 0 and then a decoded picture hash SEI message is created in the same access unit but is given TId 1. Then the marker bit will be set on the packet containing the decoded picture hash SEI message which is put in stream B. If this scheme is applied for all access units, stream A will not contain any packets with the marker bit set since these are all put in stream B. A MANE that only wants to forward stream A to a receiver would be required to perform analysis (and probably introduce delay) in order to correctly change the marker bit on the "new" last packet of the access unit.
>> 
>> Is that correct, or have I misinterpreted anything?
>> 
>> Best Regards Jonatan
>> 
>> -----Original Message-----
>> From: Wang, Ye-Kui [mailto:[email-masked]]
>> Sent: den 4 mars 2014 18:21
>> To: payload issue tracker; 
>> [email-masked]; [email-masked]; Jonatan 
>> Samuelsson
>> Cc: [email-masked]
>> Subject: RE: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
>> 
>> Hi Jonatan,
>> 
>> I do not fully understand the following example:
>> 
>> "However, as far as I understand, it is allowed for an encoder to create a  bitstream with two temporal sub-layers (every second picture with TId 0  and every second picture with TId 1) but put post-picture SEI messages for  all access units in temporal layer 1 (also for access units in which the  picture has TId 0, as a hint of the relative importance of the NAL units). For that case I guess there will be one RTP stream (corresponding to TId 0) completely without any marker bit set, right?"
>> 
>> Could you please check whether it can be clarified a bit?
>> 
>> BR, YK
>> 
>> -----Original Message-----
>> From: payload issue tracker [mailto:[email-masked]]
>> Sent: Monday, March 03, 2014 2:11 PM
>> To: [email-masked]; Wang, Ye-Kui; 
>> [email-masked]; [email-masked]
>> Cc: [email-masked]
>> Subject: Re: [payload] #10 (rtp-h265): Marker bit in H.265/HEVC
>> 
>> #10: Marker bit in H.265/HEVC
>> 
>> 
>> Comment (by [email-masked]):
>> 
>> Thanks Stefan for the comments and the example. It is exactly this kind of  cases I'm concerned about but I wouldn't characterize them as exotic. For  conversational applications it is natural to operate in *very* low delay  mode (i.e. not buffering packets before forwarding them) and having post-  picture SEI packetized in their own packets is the only option as soon as  the VCL NAL units are fragmented since FUs cannot be aggregated. Having to  introduce slices just to avoid FUs and thereby being able to aggregate SEI  messages with VCL NAL units does not sound like a compelling alternative.
>> 
>> The most reasonable solution for this case is probably to just not perform  thinning by removing the SEI messages since they are typically not very  large in the first place.
>> 
>> However, as far as I understand, it is allowed for an encoder to create a  bitstream with two temporal sub-layers (every second picture with TId 0  and every second picture with TId 1) but put post-picture SEI messages for  all access units in temporal layer 1 (also for access units in which the  picture has TId 0, as a hint of the relative importance of the NAL units).
>> For that case I guess there will be one RTP stream (corresponding to 
>> TId
>> 0) completely without any marker bit set, right?
>> 
>> --
>> -------------------------------------+-------------------------------
>> -------------------------------------+---
>> -------------------------------------+---
>> Reporter:                           |       Owner:  draft-ietf-payload-
>> [email-masked]    |  [email-masked]
>>    Type:  defect                   |      Status:  new
>> Priority:  major                    |   Milestone:
>> Component:  rtp-h265                 |     Version:  2.0
>> Severity:  -                        |  Resolution:
>> Keywords:                           |
>> -------------------------------------+-------------------------------
>> -------------------------------------+---
>> -------------------------------------+---
>> 
>> Ticket URL: 
>> <http://trac.tools.ietf.org/wg/payload/trac/ticket/10#comment:3>
>> payload <http://tools.ietf.org/payload/>
>> 
>> _______________________________________________
>> payload mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/payload
>> 
>> _______________________________________________
>> payload mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/payload
>> 
>