Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 11 Apr 2014 16:47:00 +0000

Thanks again! And sorry for not being able to reply earlier as I got stuck in some other duties (mainly the MPEG and JCT meetings last week and half).

Please see my replies inline below. Similarly, I also removed the issues where I am happy with your responses and have no further to add. Now only two items further discussed below.

> 
> 11. Section 4.1: Sequence number (SN): 16 bits
> 
> Set and used in accordance with RFC 3550.
> 
> To my understanding when using single stream transmission the Decoding 
> order is recovered soley from RTP sequence numbers and order of the 
> NALUs within each RTP payload. This usage should be mentioned here as 
> it affects the impact if anyone would inject other payloads in the 
> same RTP packet stream.
> 
> [YK]: The current draft also supports interleaved transmission, 
> wherein decoding order can be different from RTP sequence number 
> order. Decoder order number (DON) values are used for 
> de-packetization.

That is really not clear. My reading of the draft, really indicated that you removed all the interleaving support that was present in H.264's payload format and that DONs really was to support scalability.

If it is intended to support Interleaving, then the draft needs to be explicit about this support and discuss the limitations on that mechanism, including how the use of the signalling attribute to negotiate buffer capability.

[YK]: OK, I will try to make this clearer along with your suggestion above.

> 
> 12. Section 4.1: I am missing the SSRC usage for this RTP payload. In 
> SST that is the classical identification of a particular encoding of a 
> media source. But for MST the SSRC usage is different as one needs to 
> know which SSRC in which RTP sessions that are associated. I think 
> this dependency needs to be noted.
> 
> [YK]: OK. I'd appreciate if you could provide some concrete language 
> for this.

Ok, a draft for you to word smith.

synchronization source (SSRC): 32-bits

   Used to identify the source of the RTP packets. In SST mode a single SSRC will be used for all the parts of a single encoding. In MST mode, each SSRC is used for a packet stream containing a sub-set of encoding layers for a single encoding. Thus, a receive is required to correctly associate the set of SSRCs that included parts of the same encoding. See Section X.Y for further discussion of this association.

What I considered but haven't included is if more about the dependency tree needs to be expressed here, or if that is sufficient to be covered elsewhere?

[YK]: Thanks! Just to confirm: are you suggesting that in MST (note that MST would mean multi-steam transmission, using either a single RTP session or multiple RTP sessions) it is mandated that different packet streams must use different SSRC values (regardless of whether a single RTP session or multiple RTP sessions are used)? As for multiple dependency trees (of layers), I think that should be orthogonal to how they are partitioned into different packet streams, because in some scenarios it can be desirable to transmit two layers that are independent from each other in one packet stream.

BR, YK

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 11 Apr 2014 17:56:15 +0000

Thanks more! See more discussions inline below. I also removed all issues where we have settled down.

> 
> 23. Section 5: Otherwise (sprop- depack-buf-nalus  is  equal  to  0 
> for  an  RTP  stream),  the transmission order of NAL units carried in 
> the RTP stream MUST be the same as the NAL unit decoding order.
> 
> This fails to describe how you determines the NAL unit decoding order. 
> I assume in RTP sequence order and for each payload in the order the 
> NALUs are included?
> 
> [YK]: No, the NAL unit decoding order is described by the 
> de-packetization process in Section 6, which applies in all cases, 
> regardless of sprop-depack-buf-nalus being 0 or greater than 0. Yes, 
> when sprop-depack-buf-nalus is equal to 0, the order would be as you 
> assumed above. I don't see anything wrong or missing in this regard.
> However, if you think more clarification is needed anywhere, please 
> suggest and let us know where.

What might not be clear in Section 5, is what is counted as transmission order within an aggregation packet. That is also not given for
sprop-depack-buf-nalus=0 in Section 4.7.

But, you might be correct that the actual text that I am missing is the one that belongs to Section 6, on how to derive decoding order from received packets when sprop-depack-buf-nalus=0.

[YK] OK, I will try to make it clearer in Section 4.7 and Section 5 on what is counted as transmission order when sprop-depack-buf-nalus=0 and within an AP.

> 
> 24. Section 6:
> 
> When MST is in use, NAL units of all RTP streams are stored in the 
> same de-packetization buffer.
> 
> I think it needs to be clarified that this is all RTP streams from the 
> same encoder instance and media source. Not all RTP streams from 
> multiple encoder instances and media sources.
> 
> [YK]: Yes. But on other hand, in my understanding, we always refer to 
> the context of one "encoder instance and media source" in RTP payload 
> format for a particular codec, right? Is it OK that we just make this 
> generic clarification?

"for a particular codec" can be misinterpreted as meaning some other format, like H.264. Thus, I would prefer if you try to be more explicit so that it isn't misunderstood. Multi source and multi-stream (within a single media type) hasn't been that common.

OK, will try to clarify it along with your suggestion.

> 26. Section 7.1:
> 
> profile-space, profile-id: Otherwise (the RTP stream is a dependent 
> RTP stream), the following applies, with j being the value of the
> sub-layer- id parameter:
> 
> o profile_space = sub_layer_profile_space[j] o profile_id = 
> sub_layer_profile_idc[j]
> 
> The following text raises some concerns. A) These are defined as being 
> stream specific values. It is unclear how one deals with these if one 
> only want to configure the general capability within a RTP payload 
> type.
> 
> [YK]: Note that these are "elementary stream" specific values, not 
> "RTP stream" specific values. For example, if there are 3 temporal 
> sub-layers, with TemporalId equal to 0, 1, and 2, respectively, and 
> each is carried in one RTP stream, then the values of the RTP stream 
> corresponding to the temporal sub-layers with TemporalId equal to 2 
> are the property of the bitstream or elementary stream consisting of 
> all the three temporal sub-layers. And since this RTP stream is not a 
> dependent RTP stream (because no other RTP stream depends on it), the 
> general capability is assigned.

Well, I think you need to be explicit about what "stream" you mean in the text throughout the whole payload format.

[YK]: Agree. I will systematically check all instances of the "stream" wording and try to make the usages clear.

> 
> B) It is unclear if one may or are required to create different RTP 
> payload types for the different scalability layers. From my 
> perspective it is crucial that one can avoid a requirement to create 
> layer specific payload types as this can result in total consumption 
> of the available payload type space in an RTP session.
> 
> Thus, I see a need to clarify if and when one uses the above 
> assignment. And how the counter part in any signalling exchange is 
> supposed to interpret these cases.
> 
> [YK]: To me, when a spec does not disallow the use of different RTP 
> payload types, which is the case with the current spec, then the use 
> is allowed. Do you see any reason to impose an restriction on use of 
> payload types for the different RTP streams in a multi-stream 
> transmission? If not, I think we can just add a note to say that there 
> is no restriction imposed herein for the use of payload types.

No, I don't see a reason to impose a restriction. But, if something is intended to be possible it is better to be explicit so that the lazy implementor doesn't miss that it is required to be capable of handling such a configuration.

[YK]: Good, then let's add a note to say that this restriction is not in place.

> 
> 27. Section 7.1: profile-compatibility-indicator:
> 
> Isn't this a sprop parameter?
> 
> [YK]: It is the same as profile, tier, and level. And you can see in 
> later text, it is part of the media format configuration.
> 
> Also it is not clear what "j" is:
> 
> If the RTP stream is not a dependent RTP stream, the following applies 
> with j = 0..31:
> 
> o The 32 flags = general_profile_compatibility_flag[j]
> 
> Am I correct that j in this case are the integer identifying a 
> particular HEVC profile?
> 
> [YK]: That is true. This is clear from the semantics of the flag as 
> defined in the HEVC spec. We can try to add a bit more clarification 
> on this.

Thanks, a pointer is likely all that is necessary, but as someone who only read a few very specific parts of the HEVC spec, I don't know these are defined in the HEVC specification.

[YK]: OK, will make this clearer in the text.

> 
> 28. Section 7.1: profile-compatibility-indicator:
> 
> It is not clear to me that this parameter has long term applicability. 
> If one includes this, is that a promise until signaling changes to not 
> provide a stream that breaks the provided value? I wonder how one 
> deals with the use cases like AD splicing.
> One stream may ensure this, is it the role of the signalling to 
> enforce that what one says applies, or will be renegotiated in 
> signalling?
> 
> [YK]: Again, the nature and the usage of this parameter is the same as 
> profile, tier, and level. So, yes, it would be a promise when 
> provided, in either SDP offer/answer or declarative usage.

Good, can you make this clearer somehow?

[YK]: Yes, will try to make it clearer.

> 
> 29.
> 
> sub-layer-id:
> 
> This parameter MAY be used to indicate the highest allowed value of 
> TID in the stream.  When not present, the value of sub-layer-id is 
> inferred to be equal to 6.
> 
> This parameter is written in a generic form, however the O/A Section 
> indicates this to declare a property of the stream to be sent. But not 
> explicit discussion of this parameter is provided. So what is the 
> meaning of this parameter?
> 
> A) I promise that the streams I send does not use more than X 
> sub-layers
> 
> B) I require the peer sending me a stream to not use more than X 
> sub-layers.
> 
> [YK]: It is A. Would it be clear enough in your view if we add that 
> this parameter shall not be used for any other purpose than indicating 
> a property of the stream (which to me is sort of clear from its 
> current semantics)?

If it is only A, then I think it should have a sprop prefix.

[YK]: Agree. Will add a sprop prefix for it.

> 
> 30. Section 7.1: sprop-vps, sprop-sps and sprop-pps:
> 
> I think one unclarity and one that existed in H.264 also, is the 
> precedence between out-of-band and in-band parameter sets. And when 
> they start to apply. The later is especially important in cases when 
> you actually transmit a different set in a new signalling exchange 
> then what has previously been used.
> 
> Can some clarification on this be included?
> 
> [YK]: OK, we can add a clarification to state that, out-of-band 
> parameter sets can be put at the beginning of the bitstream output to 
> the decoder, while in-band parameter sets can be sent to the decoder 
> in the same way as the receiver outputs other types of NAL units from 
> the de-packetization process.

I am uncertain that what you try to say here. I interpret it as:

As soon as you get the out-of-band parameters sets they can be added to the front of the queue of NALUS that are given to the decoder. Is that what you intended?

[YK]: Exactly. Will add such a clarification (in a way that it is as clear as possible).

> 
> 31. Section 7.1: Many of the parameters. I notices that quite a lot of 
> the parameters are not explicitly defining the value range allowed for 
> the parameters. This is especially true for integer style values.
> See the many max- parameters.
> 
> [YK]: Right. We have been doing this all the time, including RFCs 
> 6184/6190. Are you saying that from now that the value range of each 
> parameter needs to be clearly specified in RTP payload formats?

>From my perspective, being explicit about what values you expect reduces the risk for error or becoming subject to an attack as you can parse according to the spec, and if the input is malformed or malicious you can give an error with a clear reason. If you are not explicit, you don't know what to expect here. Thus, I think explicit ranges or at least data types, i.e. what data and format to parse the value as.

[YK]: Agree with your reasons. In the video coding standard spec, we always clearly define a value range for each parameter unless the range is clear (e.g. for an 8-bit unsigned integer parameter). Just that we did not follow this practice earlier. I will try to come up with some reasonable value range for each parameter.

> 
> 32. Section 7.1: max-dpb: Formation error
> 
> [YK]: Sorry, I don't know what is error by comparing to the semantics 
> to that of other parameters. Could you please clarify?

Sorry, poorly worded. The lines on page 55 overflows the 72 column boundaries.

[YK]: OK. Will check to make sure this formatting error goes away.

> 
> 33. Section 7.1:
> 
> cap-parameter = tier-flag / level-id / max-lsr / max-lps / max-br
> 
> These ABNF elements are not defined. I think they need to be.
> 
> [YK]: This came from you guys. Can you please provide the missing 
> definition?

Yes, I know. Here is an proposal.

tier-flag = "tier-flag" EQ ("0" / "1")
level-id  = "level-id" EQ 1*3DIGIT  ; (0-255)
max-lsr   = "max-lsr" EQ  1*20DIGIT ; (0-18,446,744,073,709,551,615)
max-lps   = "max-lps" EQ 1*10DIGIT  ; (0-4,294,967,295)
max-br    = "max-br"  EQ 1*20DIGIT  ; (0-18,446,744,073,709,551,615)
EQ = "="
DIGIT = <RFC 5234>

This is based on my understanding that tier-flag really only is a binary flag. level-id is an unsigned 8-bit value. max-lsr and max-br clearly has the potential for being larger than 32-bit integer, thus I used a 64-bit integer. I think 4G of pixel resolution is likely sufficient.

Any errors in these assumptions?

[YK]: Thanks a lot! I think your assumptions are correct. Will add these syntax definitions.

> 
> 35. Section 7.2.2:
> 
> Informative note:  The above parameters apply for any stream sent by a 
> declaring entity with the same configuration; i.e. they are dependent 
> on their source.  Rather than being bound to the payload type, the 
> values may have to be applied to another payload type when being sent, 
> as they apply for the configuration.
> 
> I find this confusing. I don't know if it is a typo and should say 
> "independent of their source", or if it is the definition of source 
> here that is wrong.
> 
> It might be that the paragraph before the above: o  The parameters 
> sprop-depack-buf-nalus and sprop-depack-buf-bytes describe the 
> properties of the RTP stream that the offerer or the answerer is 
> sending for the media format configuration.  This differs from the 
> normal usage of the Offer/Answer parameters: normally such parameters 
> declare the properties of the stream that the offerer or the answerer 
> is able to receive.  When dealing with HEVC, the offerer assumes that 
> the answerer will be able to receive media encoded using the 
> configuration being offered.
> 
> actually needs to separate between end point specific configurations 
> that apply to the whole RTP session through the payload type or if is 
> on specific media encoding basis and thus needs to be connected to the 
> SSRC(s) that carries the particular media encoding.
> 
> [YK]: It seems to be a good catch. The same wording was inherited from 
> RFC 6184, and appeared in RFC 3984 as well, for which you were a 
> co-author. Could you authors of RFC 3984 (e.g. Stephan, Miska,
> Magnus) try to help figure out what was the actual intent when this 
> wording was firstly introduced into RFC 3984?

What, you are holding me responsible for what I wrote 10 years ago? ;-)

Considering this is SDP O/A, and looking at it again. I think it refer to that an SDP agent sending a Offer or Answer with sprop-depack-buf-nalus and sprop-depack-buf-bytes included, are responsible to ensure that any HEVC encoding and its RTP packetization conforms to the declared parameters. This applies to each encoding independently, even if multiple encodings are transmitted.

Thus, I think this text would be clearer if one adds "endpoint" after source.

[YK]: Thanks a lot! So holding you responsible for what you wrote 10 years ago did help ;-)

> 
> 36. Section 7.2.2:
> 
> min_spatial_segmentation_idc  equal  to  or  larger  than  dec- 
> parallel-cap.spatial-seg-idc of the capability point.  A stream that 
> is sent based on choosing a capability point with parallel tool
> type    't'    from    dec-parallel-cap    MUST    have 
> entropy_coding_sync_enabled_flag     equal     to     0     and 
> min_spatial_segmentation_idc  equal  to  or  larger  than  dec- 
> parallel-cap.spatial-seg-idc of the capability point.
> 
> It looks like someone managed to turn this text into straight left and 
> right columns, this is not allowed in IETF, we use ragged rights.
> 
> [YK]: I guess you meant to align text only to the left margin, not to 
> both left and right margins? If that is the case, we actually did this 
> in all the places in the Word based template for preparing of the 
> draft. Do you see any issues in other places as well? Can you please 
> let us know some specific line for which the style should be changed?

Correct, I have seen this in some few places.

Lines:
33
58, for three lines
262
285
299, for 3 lines
376
etc.

3275, for 5-lines

Search for ".   " and you will find a large number of these issues.

[YK]: Thanks! I see. I will change all alignments to ragged right alignment.

> 
> 37. Section 7.2.2:
> 
> o  An offerer has to include the size of the de-packetization buffer, 
> sprop-depack-buf-bytes,  and  sprop-depack-buf-nalus,  in the  offer 
> for  an  interleaved  HEVC  stream  or  for  the  MST transmission 
> mode.  To enable the offerer and answerer to inform each  other about  
> their  capabilities  for  de-packetization buffering in receiving 
> streams, both parties are RECOMMENDED to include depack-buf-cap.  For 
> interleaved streams or in MST, it is also RECOMMENDED to consider 
> offering multiple payload types with different buffering requirements 
> when the capabilities of the receiver are unknown.
> 
> This paragraph contains two mentions of "interleaved", the only two in 
> the whole specification. I assume a copy and paste from RFC 6184 that 
> wasn't converted properly.
> 
> [YK]: No they are intentionally left here, as interleaved 
> packetization is allowed. Maybe we can add one sentence or two to make 
> this more explicit.

As said earlier, if this is a supported feature of the payload format, then you need to discuss it explicitly.

[YK]: Agreed.

> 43. Section 7.2.2:
> 
> sendonly --+ answer: recvonly, recv-sub-layer-id --+  | recvonly w/o 
> recv-sub-layer-id --+  |  | answer: sendrecv, recv-sub-layer-id --+
> |  |  | sendrecv w/o recv-sub-layer-id --+  |  |  |  | |  |  |  |  |
> profile-space                      C  X  C  X  P profile-id
> C  X  C  X  P tier-flag                          C  X  C  X  P 
> level-id                           C  X  C  X  P
> 
> C: configuration for sending and receiving streams
> 
> I don't get this usage or definition of C for these parameters.
> 
> So these parameters are receiver capabilities, and they needs to be 
> symmetrically used in the signalling, however the actually used values 
> in the transmission direction from an O/A agent only needs to be 
> within the receivers provided parameters. Thus, they are not a 
> "configuration" for the sending direction.
> 
> [YK]: These are configuration parameters, meaning indicating both what 
> is to be sent and what is to be received for sendrecv and must be used 
> symmetrically, and only the latter for recvonly.

This is at a least wrong for the level-id which a peer agent in an O/A may downgrade. The rest I agree the payload format itself requires to be kept in answer, or the corresponding PT dropped.

[YK]: OK, level-id is downgradable but it is still a configuration, but a special configuration that can be downgraded. I'd suggest we keep using the "C" denotation but adding a note to repeat what is said earlier that level is downgradable, because using any other denotation is not correct. Please let me know if you think this is not OK, and if so what kind of changes would you propose.

> 
> Also I really don't get the X's in the answer: recvonly, 
> recv-sub-layer-id, and answer: sendrecv, recv-sub-layer-id. The 
> answering agent is going to receive stream in both these cases, thus 
> it needs to declare the payload type, and these parameters MUST be 
> included (unless defaulted). So why is they marked with X?
> 
> [YK]: When recv-sub-layer-id is included in an SDP answer, it means 
> that the answer chooses one of the multiple operation points 
> corresponding to the offered or declared sub-layers in the sprop-vps, 
> and in this case there is no need to repeat the information of 
> profile, level etc. in the SDP, thus there are marked with X (MUST NOT 
> be present).

This appears very strange to me. The Offerer includes a VPS that contains options for what it can send. The answering party, i.e. which will receive this stream needs in accordance to SDP O/A rules declare which PTs it can receive. This PT must be corresponding to the one that the Offerer proposed to send and in which the VPS was included.

For me this implies the following:

1. The parameters indicating the basic configuraiton of the payload type, i.e profile, level ,tier etc. MUST be present in the answer with the same values as in the offer it responds to. Otherwise the PT matching rule fails.

2. In addition the PT to receive on MUST be declared as existing to ensure that any middleboxes doesn't block it. Thus, explicit indication of what will go in it is really recommended, especially considering i all the other cases that information will be explicit here. Thus, I think the X's are plain wrong. Or I am still misunderstanding something here.

[YK]: Let's use an example, which should help. Let's say the offer SDP includes a VPS that provides two operation points, 720p@15fps and 720p@30fps, corresponding to recv-sub-layer-id 0 and 1, respectively. The profile, tier, level, and so on for each operation point are signalled by the VPS. If recv-sub-layer-id=0 is included in an SDP answer, then that means the answerer chooses to operate at 720p@15fps. In this case, the profile, tier, level, and so on for the chosen operation point does not need to be explicitly sent again in the SDP, as it is clear what they are (they are signalled in the VPS in the SDP offer). So there is no ambiguity here. Are you suggesting that in some use cases even there is no ambiguity the configuration information must always be explicitly signalled (i.e. values that are derivable)? If yes, could you please explain in what use cases and why. Note that we had the same mechanism specified for SVC in RFC 6190. Also, on PT use, we have allowed the use of a different PT value in an SDP answer than the matching PT value in the offer since at least RFC 3984 as long as the PT matching is clear.

> 
> 44. Section 7.2.2:
> 
> sendonly --+ answer: recvonly, recv-sub-layer-id --+  | recvonly w/o 
> recv-sub-layer-id --+  |  | answer: sendrecv, recv-sub-layer-id --+
> |  |  | sendrecv w/o recv-sub-layer-id --+  |  |  |  | |  |  |  |  |
> interop-constraints                C  X  C  X  P 
> profile-compatibility-indicator    C  X  C  X  P
> 
> I don't understand how these parameters really are used for receiver 
> capability declarations, so I don't understand why the Cs are C and 
> not P.
> 
> [YK]: As explained earlier, these are also parts of the media type 
> configuration, same as profile, tier, and level, and their usage are 
> the same as those too.

First of all these parameters are not included as the ones that have to be symmetrically provided. I also think they could be SDP agent specifically given. But, that can be discussed. But, according to the current text, I fail to see that these needs to be handled as the profile-id, level-id, etc.

The only constraint I find for these two flags applies under multicast.

Either they need text, or these C's need to be changed.

[YK]: Maybe you were initially reading version -01 and did not check version -02 on this part. In version -02, these parameters are included in the list of configuration parameters that are specified to be used symmetrically.

BR, YK

---

From: Stephan Wenger <[email-masked]>
Date: Sun, 13 Apr 2014 22:20:03 +0000

Hi all, especially Magnus,

Magnus, thanks again for your careful review.

Below inline some comments on Magnus issues #17 through #22, PACI-related.
 All other issues have been removed from this email.

Regards,
Stephan

On 28.2.14, 7:47, "Magnus Westerlund" <[email-masked]>
wrote:

>[Š]
>
>17. Section 4.9:
>      0                   1                   2                   3
>       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>      |                          RTP Header                           |
>      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>      |F| PACI=50   |  LayerId  | TID |A|    Type   | PHSsize |F0..2|X|
>      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>      |        Payload Header Extension Structure (PHES)              |
>      |=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=|
>      |                                                               |
>      |                  PACI payload: NAL unit                       |
>      |                   . . .                                       |
>      |                                                               |
>      |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>      |                               :...OPTIONAL RTP padding        |
>      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
>
>I wonder if one shouldn't mark the first 16 bits after the header as the
>"payload header" and instead separate out the configuration of those for
>this payload format. That would also resolve the next issue.

Yes, I agree.  In fact, it¹s kind of odd that section 4.9 is the first
instance where we have a figure explaining the (main) payload header
itself.  Of course, we have done so in textual form in some detail before,
but not in a figure.

My current thinking is that the right place for such a figure would be
section 4.3, as these 16 bits are the same for all four packet types we
have.  And keep the section 4.9 figure as is, plus/minus the changes
identified in issues #18 and #19 below.

>
>18. Section 4.9:
>
>   PACI: 6 bits
>      Indicates a PACI, and must be 50.
>
>As this field is called Type in the other definition, why not call it
>Type: 6 bits
>    Indicates a PACI, and must be 50.
>
>And to clarify the confusion with the PACI payload type, that Type field
>should be called IType or something like that

Both suggestions make sense (regardless of the doc structure fix of issue
#17 above) and we will implement them.

>
>19. Section 4.9:
>   PHSsize: 5 bits
>      Indicates the total length of the PHES.  The value is limited to
>      be less than or equal to 32 octets, to simplify encoder design
>      for MTU size matching.
>
>   F0..2: 3 bits
>      Each of the three bits indicate, when set, the presence of an
>      optional field (or set of fields) in the PHES.
>
>   X: 1 bit
>      The X bit, when set, indicates the presence of another octet
>      consisting of seven flags and another X bit, each of the seven
>      flags indicating the presence of more PHES fields (for future
>      extensions).
>
>The motivation of the PHSSize of limiting this to 32 octets is not
>fulfilled. As the RTP payload still are variable size due to the X bit,
>which adds one or more bytes. Thus if the goal is to make clear that
>this will never be more than 32 bytes, than I think the bytes that X
>signals must be included in the 32-bytes for the PHES.

It was unwise of me to call this extension bit ³X².  It is easily confused
with the X bit in the RTP header.  We should replace the section 4.9 ³X²
with another suitable character.
The reminder of my answer depends on how you interpret ³X²: the PHS
extension X, or the RTP header X.

In case you meant PHS extension ³X²:
You are correct, and the fix is easy: We add a requirement that PHSsize
shall include the extension flag octets (whose presence is indicated by
those X bits).

In case you meant RTP header ³X²:
The bug fix as above still needed.  In addition, we could also add
somewhere in section 4.1 (RTP header usage) a requirement that the total
size of the RTP header, plus RTP header extension, plus the payload header
and all its extensions shall not exceed a certain number.  That number is
not easy to calculate, as it depends on the use of PACI (and its extension
mechanism), but also FU, aggregation, DON or no DON, and so on.  I¹m not
in favor of doing that.

However, what might be a good idea is to add an informative note into
section 4.1 to the extent that a good encoder fills up the MTU whenever it
can so to keep packetization overhead low, but in order to do so, it has
to be able to predict or know the size of the payload header, which is
highly variable due to syntax options defined in this specification, but
also through the potential presence of RTP header extensions.

Unrelated to the above:
I can imagine cases where we would need only a single bit to signal
something.  In that case, it would be kind of odd to use a bit to signal
the presence of another bit.  I don¹t think that needs to be specified in
any detail, except by saying that the extension signalized by an F bit set
to 1 can have a length of zero octets.  Good idea?

>
>20. Section 4.9: Forward compability for PHES.
>
>The content of the PHES field is determined using the F0..2 field and in
>the future the octet(s) added using the X field, where each set bit
>indicates what fields are included. However, this definition have some
>severe implications regarding future compatibility. As each bit can
>represent an unknown amount of fields a first gen decoder can't know how
>many bits each of the bits represents. This may not matter much for the
>first gen, as it only knows of the PHES data defined in this version. To
>be able to decode these fields only a single thing is missing, an
>assurance that the first defined data MUST be first in the PHES data
>section when F0 is set.
>
>For a third generation implementation there is another implication. To
>know where the F2 PHES payload starts it MUST know also what F0 and F1
>means in amount of fixed field data values.
>
>I am not that happy for this forward compatibility limiations. But, it
>can work, but the rules for how to make it work MUST be explictely
>stated now.
>
>1) The fields added for a particular FX bit MUST be fixed in length and
>not depend on what other FX bits are set.
>2) The FX bits must be assigned in order
>3) An implementation that supports FX for any value will be required to
>understand what fields are added and their size for all FX bits that are
>X or less.

Again, I agree with this bug report and with the proposed fix.  I¹m not
sure that it is best placed in normative language, though (as your
capitalization of MUST suggests).  The three points made above are
reminders for future spec developers, and not protocol constraints (well,
mostly not).  In video codec standards land, tons of such constraints are
made in early versions of the spec, and never documented outside of
meeting reports.  Luckily, there is sufficient institutional memory in the
organization that they are rarely forgotten.  In the IETF in general and
in the payload WG in particular, I¹m not so sure about that institutional
memory.
Therefore, I propose to add your 3 constraints in informative language.
See issue #21 below.

If you have a to-do list for the next rev of the RTP payload how-to: I
think that a section "Extension Mechanisms -- Gremlins here² would be
useful.  Also, if there were a generic ftp payload format syntax for
extensions, I think we would have used it (even if its overhead would have
been larger than the one we developed here).  Perhaps someone with enough
time on hand could write something like that up, for future use?  In video
codec land, we did this once for H.263+ SEI messages, and re-used it
thereafter basically forever.

>
>21. Section 4.9: I think the motivation for the PACI is missing. What
>are the motivation for providing this information in the PACI and what
>would any RTP endpoint or middlebox use the information for.

I propose to add as second paragraph of 4.9:
³
The basic payload header specified in this memo is intentionally limited
to the 16 bits of the NAL unit header so to keep the packetization
overhead to a minimum.  However, cases have been identified where it is
advisable to include control information in an easily accessible position
in the packet header, despite the additional overhead. One such control
information is the Temporal Scalability Control Information as specified
in section 4.10/4.10.1 below.  However, other types of control information
may be identified in the future.  For this reason, an extensible syntax is
included that allows future versions of this specification, or companion
specifications, to defined their own control information and include it
into the payload header.

When a spec developer devises a new syntax that takes advantage of the
PACI extension mechanism, he/she must follow the constraints listed below;
otherwise the extension mechanism may break.  <points 1 through 3 of issue
#20². 
³

>
>22. Section 4.10:
>
>Can you please provide a name for the functionality set the F0 bit
>represents so that we can use a name to reference it?

³Temporal Scalability Control Information².
That should probably also be the section head of 4.10, or we could keep
4.10, saying that there is only a single extension defined in this memo,
and then 4.10.1 ³Temporal Scalability Control Information².  Preferences,
anyone?

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 14 Apr 2014 11:49:21 +0200

Hi,

Response below to the second issue.

/Magnus

On 2014-04-11 18:47, Wang, Ye-Kui wrote:
>> 
>> 12. Section 4.1: I am missing the SSRC usage for this RTP payload.
>> In SST that is the classical identification of a particular
>> encoding of a media source. But for MST the SSRC usage is different
>> as one needs to know which SSRC in which RTP sessions that are
>> associated. I think this dependency needs to be noted.
>> 
>> [YK]: OK. I'd appreciate if you could provide some concrete
>> language for this.
> 
> Used to identify the source of the RTP packets. In SST mode a single
> SSRC will be used for all the parts of a single encoding. In MST
> mode, each SSRC is used for a packet stream containing a sub-set of
> encoding layers for a single encoding. Thus, a receive is required to
> correctly associate the set of SSRCs that included parts of the same
> encoding. See Section X.Y for further discussion of this
> association.
> 
> What I considered but haven't included is if more about the
> dependency tree needs to be expressed here, or if that is sufficient
> to be covered elsewhere?
> 
> [YK]: Thanks! Just to confirm: are you suggesting that in MST (note
> that MST would mean multi-steam transmission, using either a single
> RTP session or multiple RTP sessions) it is mandated that different
> packet streams must use different SSRC values (regardless of whether
> a single RTP session or multiple RTP sessions are used)?

I think so, as people are likely to do MST in a single RTP session, the
requirements and hooks for such a solution are needed. Thus, the same
SSRC in different RTP sessions are not a sufficient solution. Requiring
a solution that can deal with different SSRCs, thus ensure that even if
the same SSRC is used different RTP sessions, that is just a unlikely
event, not the basis for the solution.

 As for
> multiple dependency trees (of layers), I think that should be
> orthogonal to how they are partitioned into different packet streams,
> because in some scenarios it can be desirable to transmit two layers
> that are independent from each other in one packet stream.

Yes, agree with that. However, I think my thoughts around multiple
dependency trees arise from that these can be independent trees and thus
individual sets of representation where either of the trees can be
independently decoded. Thus, this gets into "simulcast" land.

cheers

Magnus Westerlund

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Mon, 14 Apr 2014 17:36:02 +0000

Thanks again! Your suggestion on using different SSRC values for MST is good to me. Cool - I think we are done with this thread - of course I still need to implement them into text.

BR, YK

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Monday, April 14, 2014 2:49 AM
To: Wang, Ye-Kui; [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

Hi,

/Magnus

On 2014-04-11 18:47, Wang, Ye-Kui wrote:
>> In SST that is the classical identification of a particular encoding 
>> of a media source. But for MST the SSRC usage is different as one 
>> needs to know which SSRC in which RTP sessions that are associated. I 
>> think this dependency needs to be noted.
>> 
> SSRC will be used for all the parts of a single encoding. In MST mode, 
> each SSRC is used for a packet stream containing a sub-set of encoding 
> layers for a single encoding. Thus, a receive is required to correctly 
> associate the set of SSRCs that included parts of the same encoding. 
> See Section X.Y for further discussion of this association.
> 
> What I considered but haven't included is if more about the dependency 
> tree needs to be expressed here, or if that is sufficient to be 
> covered elsewhere?
> 
> packet streams must use different SSRC values (regardless of whether a 
> single RTP session or multiple RTP sessions are used)?

I think so, as people are likely to do MST in a single RTP session, the requirements and hooks for such a solution are needed. Thus, the same SSRC in different RTP sessions are not a sufficient solution. Requiring a solution that can deal with different SSRCs, thus ensure that even if the same SSRC is used different RTP sessions, that is just a unlikely event, not the basis for the solution.

 As for
> that are independent from each other in one packet stream.

Yes, agree with that. However, I think my thoughts around multiple dependency trees arise from that these can be independent trees and thus individual sets of representation where either of the trees can be independently decoded. Thus, this gets into "simulcast" land.

cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 25 Apr 2014 11:35:48 +0200

Hi,

Sorry for taking my time to respond to this. I have removed the issues
where it appeared that we had a resolution.

Unfortunately I think I have discovered a new issue, see the last part
of this email.

On 2014-04-11 19:56, Wang, Ye-Kui wrote:

>> 43. Section 7.2.2:
>> 
>> sendonly --+ answer: recvonly, recv-sub-layer-id --+  | recvonly
>> w/o recv-sub-layer-id --+  |  | answer: sendrecv, recv-sub-layer-id
>> --+ |  |  | sendrecv w/o recv-sub-layer-id --+  |  |  |  | |  |  |
>> |  | profile-space                      C  X  C  X  P profile-id C
>> X  C  X  P tier-flag                          C  X  C  X  P 
>> level-id                           C  X  C  X  P
>> 
>>  symmetrically used in the signalling, however the actually used
>> values in the transmission direction from an O/A agent only needs
>> to be within the receivers provided parameters. Thus, they are not
>> a "configuration" for the sending direction.
>> 
>> [YK]: These are configuration parameters, meaning indicating both
>> what is to be sent and what is to be received for sendrecv and must
>> be used symmetrically, and only the latter for recvonly.
> 
> This is at a least wrong for the level-id which a peer agent in an
> O/A may downgrade. The rest I agree the payload format itself
> requires to be kept in answer, or the corresponding PT dropped.
> 
> [YK]: OK, level-id is downgradable but it is still a configuration,
> but a special configuration that can be downgraded. I'd suggest we
> keep using the "C" denotation but adding a note to repeat what is
> said earlier that level is downgradable, because using any other
> denotation is not correct. Please let me know if you think this is
> not OK, and if so what kind of changes would you propose.

I think that is okay. But, considering that the level is here not
combined, wouldn't make sense to use some other property identifier for it?

>> answering agent is going to receive stream in both these cases,
>> thus it needs to declare the payload type, and these parameters
>> MUST be included (unless defaulted). So why is they marked with X?
>> 
>> corresponding to the offered or declared sub-layers in the
>> sprop-vps, and in this case there is no need to repeat the
>> information of profile, level etc. in the SDP, thus there are
>> marked with X (MUST NOT be present).
> 
> This appears very strange to me. The Offerer includes a VPS that
> contains options for what it can send. The answering party, i.e.
> which will receive this stream needs in accordance to SDP O/A rules
> declare which PTs it can receive. This PT must be corresponding to
> the one that the Offerer proposed to send and in which the VPS was
> included.
> 
> 1. The parameters indicating the basic configuraiton of the payload
> type, i.e profile, level ,tier etc. MUST be present in the answer
> with the same values as in the offer it responds to. Otherwise the PT
> matching rule fails.
> 
> 2. In addition the PT to receive on MUST be declared as existing to
> ensure that any middleboxes doesn't block it. Thus, explicit
> indication of what will go in it is really recommended, especially
> considering i all the other cases that information will be explicit
> here. Thus, I think the X's are plain wrong. Or I am still
> misunderstanding something here.
> 
> [YK]: Let's use an example, which should help. Let's say the offer
> SDP includes a VPS that provides two operation points, 720p@15fps and
> 720p@30fps, corresponding to recv-sub-layer-id 0 and 1, respectively.
> The profile, tier, level, and so on for each operation point are
> signalled by the VPS. If recv-sub-layer-id=0 is included in an SDP
> answer, then that means the answerer chooses to operate at
> 720p@15fps. In this case, the profile, tier, level, and so on for the
> chosen operation point does not need to be explicitly sent again in
> the SDP, as it is clear what they are (they are signalled in the VPS
> in the SDP offer). So there is no ambiguity here. Are you suggesting
> that in some use cases even there is no ambiguity the configuration
> information must always be explicitly signalled (i.e. values that are
> derivable)? If yes, could you please explain in what use cases and
> why. Note that we had the same mechanism specified for SVC in RFC
> 6190. Also, on PT use, we have allowed the use of a different PT
> value in an SDP answer than the matching PT value in the offer since
> at least RFC 3984 as long as the PT matching is clear.

Okay, I can now put the finger on what troubles me with this. I think
this is violation or at least an omission of the principle of SDP
Offer/Answer. The reason is that you substitute the answering explicitly
declaring its receiving capabilities with a reference to a particular
instantiation of a video sequence parameter set that falls within the
receiver's capabilities. Thus, the offerer does not know the receiver's
actually capabilities if it needs to reconfigure the encoder, or in case
the RTP payload type is going to be used by multiple media streams and
possibly different encoder instances.

In addition it creates an issue for any middlebox that is on the
signalling path that it keeps track and state for what was in the offer
to know what the answer means.

Thus, I think it is important that the answerer doesn't treat this
situation differently, and instead declare its capabilities as normally
in answer. The recv-sub-layer-id parameter is an optional parameter and
it is also stream specific, while the profile, level, etc are
generically applicable to the RTP payload type do applies on RTP session
level.

To conclude, the recv-sub-layer-id is fine as a stream specific
selection mechanism, but does not replace the capability declaration the
regular parameters perform.

> 
>> 
>> 44. Section 7.2.2:
>> 
>> sendonly --+ answer: recvonly, recv-sub-layer-id --+  | recvonly
>> w/o recv-sub-layer-id --+  |  | answer: sendrecv, recv-sub-layer-id
>> --+ |  |  | sendrecv w/o recv-sub-layer-id --+  |  |  |  | |  |  |
>> |  | interop-constraints                C  X  C  X  P 
>> I don't understand how these parameters really are used for
>> receiver capability declarations, so I don't understand why the Cs
>> are C and not P.
>> 
>>  configuration, same as profile, tier, and level, and their usage
>> are the same as those too.
> 
> First of all these parameters are not included as the ones that have
> to be symmetrically provided. I also think they could be SDP agent
> specifically given. But, that can be discussed. But, according to the
> current text, I fail to see that these needs to be handled as the
> profile-id, level-id, etc.
> 
> [YK]: Maybe you were initially reading version -01 and did not check
> version -02 on this part. In version -02, these parameters are
> included in the list of configuration parameters that are specified
> to be used symmetrically.

Okay, I had missed that change in the -02 version.

After having given this some thought and re-read the definitions and
relevant SDP O/A parts I think I have found a new issue.

This issue is related to the interpretation of  "interop-constraints",
and "profile-compatibility-indicator". It is not clear in which
direction they should be interpreted in the various cases. Are they
constraints on what the receiver/decoder accept or constraints the
encoder promise to fulfil or both?

I think the "profile-compatibility-indicator" is an excellent case to
show that neither of the alternative appears to work or provide any
benefit. This example will use a future profile that I call Baseline,
which is a sub-set of Main.

Receiver constraints:

A sends it offers with a RTP payload type (97) which is Main but with
profile-compatibility-indicator indicating Main and Baseline. This means
the offerer prefers to do main but knows of baseline. Answerer is
baseline only. Thus it detects a profile not supported, but
compatibility indicates support. However, due to the other payload type
rules, the answerer must reject PT=97 as it can't answer with
97=Baseline. Nor can it reject 97 and and 98 which is baseline, and only
use 97 in answerer to offerer direction. And only receive using 98.
Especially not if 97 was the only PT offered, then you have no
compatibility.

Instead, to my understanding to achieve receiver constraints the offerer
must include one PT per profile it supports, the
profile-compatibility-indicator is useless in this case.

Forcing the answerer to copy it and be forced to have a the same
receiver constraints is just useless.

Sender constraints:

If this is a sender constraint, then it actually tells something about
the sender's capability of providing something that matches more
profiles than what it is willing to receive. However, the answerer if it
doesn't support the any of the PTs and their profile-id values then it
anyway have to refuse the PT, even if it could receive the stream from
the offerer. Instead, also here the offerer needs to indicate all
profiles it can encoded for and offer them. Which implies also decoding
capability. In addition, if this is a sender constraint, using C as
classification (i.e. must copy) in answer, if the encoder in the
answerer supports the payload types profile-id but, can't ensure or know
about all the profile-compatibility flags, then it has to refuse the
payload type despite it supporting it.

Result, forcing C on a sender constraint interpretation only results in
failures and no benefit.

Receiver and Sender constraints.

To my understanding such an interpretation only forces the case of more
rejections, if the answerer doesn't have an equally capable encoder and
decoder as the offerer and can match both sets of constraints. In
addition the compatibility has not provided any benefit for the two O/A
agents.

So, the question becomes have I misunderstood something about how you
intended to be able to use this parameter?

If I haven't I think these parameters usage is seriously in question. If
one would like to use it then the must not alter constraint for profile
becomes a serious issue. If one would remove that then I see that using
"profile-compatibility-indicator" could be possibly beneficial. But, I
haven't done a full out design of how to resolve the PT matching and
deal with asymmetric usage within the session.

The same type of analysis and clarification on usage needs to happen for
the "interop-constraints".

Then we have the question of how one shall interpret and deal with
unknown flag values. These needs to cause only failures when required
for ensuring video compatibility. Not cause negotiation failures for
unknown cases that doesn't affect video interoperability.

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 25 Apr 2014 11:56:54 +0200

> shall include the extension flag octets (whose presence is indicated by

I meant this interpretation of X. I am fine with that solution limiting
the PACI content including the Flag octets to 32 octets.

> to 1 can have a length of zero octets.  Good idea?

Yes, I think that is fine.

> thereafter basically forever.

I haven't created such a list yet. I guess it might be time to start one.

> otherwise the extension mechanism may break.  <points 1 through 3 of issue

Appears reasonable.

> and then 4.10.1 ³Temporal Scalability Control Information².  Preferences,

Works for me, no preference.

Cheers

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Fri, 25 Apr 2014 17:56:27 +0000

Thanks! Please see inline below.

For the third item, which is also the last one, let me also put my response here, to draw more attention from interested people in this list:

[YK]: They are meant for Receiver and Sender constraints. What we have now is similar to what we have in RFC 6184 and RFC 6190, with the essential difference been that in AVC there is only one byte of such constraints, while in HEVC there are more bytes of such constraints. I think what you were thinking of is some optimization. Considering the urgent situation that the RFC is urgent needed by the relevant industries for HEVC deployment, I personally would prefer to go ahead without trying to make more optimization here, but let it be similar as in RFC 6184 and RFC 6190. However, if the group thinks that we should do more optimization here, then we can try it, but this needs careful study and investigation.

>> recv-sub-layer-id --+  |  | answer: sendrecv, recv-sub-layer-id
>> values in the transmission direction from an O/A agent only needs to 
>> be within the receivers provided parameters. Thus, they are not a 
> This is at a least wrong for the level-id which a peer agent in an O/A 
> may downgrade. The rest I agree the payload format itself requires to 
> be kept in answer, or the corresponding PT dropped.
> 
> keep using the "C" denotation but adding a note to repeat what is said 
> earlier that level is downgradable, because using any other denotation 
> is not correct. Please let me know if you think this is not OK, and if 
> so what kind of changes would you propose.

I think that is okay. But, considering that the level is here not combined, wouldn't make sense to use some other property identifier for it?

[YK]: Could you please clarify what does it mean by "not combined", and by "to use some other property identifier"?

>> profile, level etc. in the SDP, thus there are marked with X (MUST 
>> NOT be present).
> 
> declare which PTs it can receive. This PT must be corresponding to the 
> one that the Offerer proposed to send and in which the VPS was 
> included.
> 
> type, i.e profile, level ,tier etc. MUST be present in the answer with 
> the same values as in the offer it responds to. Otherwise the PT 
> matching rule fails.
> 
> [YK]: Let's use an example, which should help. Let's say the offer SDP 
> includes a VPS that provides two operation points, 720p@15fps and 
> answer, then that means the answerer chooses to operate at 720p@15fps. 
> In this case, the profile, tier, level, and so on for the chosen 
> operation point does not need to be explicitly sent again in the SDP, 
> as it is clear what they are (they are signalled in the VPS in the SDP 
> offer). So there is no ambiguity here. Are you suggesting that in some 
> use cases even there is no ambiguity the configuration information 
> must always be explicitly signalled (i.e. values that are derivable)? 
> If yes, could you please explain in what use cases and why. Note that 
> we had the same mechanism specified for SVC in RFC 6190. Also, on PT 
> use, we have allowed the use of a different PT value in an SDP answer 
> than the matching PT value in the offer since at least RFC 3984 as 
> long as the PT matching is clear.

Okay, I can now put the finger on what troubles me with this. I think this is violation or at least an omission of the principle of SDP Offer/Answer. The reason is that you substitute the answering explicitly declaring its receiving capabilities with a reference to a particular instantiation of a video sequence parameter set that falls within the receiver's capabilities. Thus, the offerer does not know the receiver's actually capabilities if it needs to reconfigure the encoder, or in case the RTP payload type is going to be used by multiple media streams and possibly different encoder instances.

In addition it creates an issue for any middlebox that is on the signalling path that it keeps track and state for what was in the offer to know what the answer means.

Thus, I think it is important that the answerer doesn't treat this situation differently, and instead declare its capabilities as normally in answer. The recv-sub-layer-id parameter is an optional parameter and it is also stream specific, while the profile, level, etc are generically applicable to the RTP payload type do applies on RTP session level.

To conclude, the recv-sub-layer-id is fine as a stream specific selection mechanism, but does not replace the capability declaration the regular parameters perform.

[YK]: OK, we will try to make this change.

>> recv-sub-layer-id --+  |  | answer: sendrecv, recv-sub-layer-id
> included in the list of configuration parameters that are specified to 
> be used symmetrically.

After having given this some thought and re-read the definitions and relevant SDP O/A parts I think I have found a new issue.

This issue is related to the interpretation of  "interop-constraints", and "profile-compatibility-indicator". It is not clear in which direction they should be interpreted in the various cases. Are they constraints on what the receiver/decoder accept or constraints the encoder promise to fulfil or both?

I think the "profile-compatibility-indicator" is an excellent case to show that neither of the alternative appears to work or provide any benefit. This example will use a future profile that I call Baseline, which is a sub-set of Main.

Receiver constraints:

A sends it offers with a RTP payload type (97) which is Main but with profile-compatibility-indicator indicating Main and Baseline. This means the offerer prefers to do main but knows of baseline. Answerer is baseline only. Thus it detects a profile not supported, but compatibility indicates support. However, due to the other payload type rules, the answerer must reject PT=97 as it can't answer with 97=Baseline. Nor can it reject 97 and and 98 which is baseline, and only use 97 in answerer to offerer direction. And only receive using 98.
Especially not if 97 was the only PT offered, then you have no compatibility.

Instead, to my understanding to achieve receiver constraints the offerer must include one PT per profile it supports, the profile-compatibility-indicator is useless in this case.

Forcing the answerer to copy it and be forced to have a the same receiver constraints is just useless.

Sender constraints:

If this is a sender constraint, then it actually tells something about the sender's capability of providing something that matches more profiles than what it is willing to receive. However, the answerer if it doesn't support the any of the PTs and their profile-id values then it anyway have to refuse the PT, even if it could receive the stream from the offerer. Instead, also here the offerer needs to indicate all profiles it can encoded for and offer them. Which implies also decoding capability. In addition, if this is a sender constraint, using C as classification (i.e. must copy) in answer, if the encoder in the answerer supports the payload types profile-id but, can't ensure or know about all the profile-compatibility flags, then it has to refuse the payload type despite it supporting it.

Result, forcing C on a sender constraint interpretation only results in failures and no benefit.

To my understanding such an interpretation only forces the case of more rejections, if the answerer doesn't have an equally capable encoder and decoder as the offerer and can match both sets of constraints. In addition the compatibility has not provided any benefit for the two O/A agents.

So, the question becomes have I misunderstood something about how you intended to be able to use this parameter?

If I haven't I think these parameters usage is seriously in question. If one would like to use it then the must not alter constraint for profile becomes a serious issue. If one would remove that then I see that using "profile-compatibility-indicator" could be possibly beneficial. But, I haven't done a full out design of how to resolve the PT matching and deal with asymmetric usage within the session.

The same type of analysis and clarification on usage needs to happen for the "interop-constraints".

Then we have the question of how one shall interpret and deal with unknown flag values. These needs to cause only failures when required for ensuring video compatibility. Not cause negotiation failures for unknown cases that doesn't affect video interoperability.

[YK]: They are meant for Receiver and Sender constraints. What we have now is similar to what we have in RFC 6184 and RFC 6190, with the essential difference been that in AVC there is only one byte of such constraints, while in HEVC there are more bytes of such constraints. I think what you were thinking of is some optimization. Considering the urgent situation that the RFC is urgent needed by the relevant industries for HEVC deployment, I personally would prefer to go ahead without trying to make more optimization here, but let it be similar as in RFC 6184 and RFC 6190. However, if the group thinks that we should do more optimization here, then we can try it, but this needs careful study and investigation.

BR, YK

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 28 Apr 2014 08:50:32 +0200

On 2014-04-25 19:56, Wang, Ye-Kui wrote:
> For the third item, which is also the last one, let me also put my
> response here, to draw more attention from interested people in this
> list:
> 
> [YK]: They are meant for Receiver and Sender constraints. What we
> have now is similar to what we have in RFC 6184 and RFC 6190, with
> the essential difference been that in AVC there is only one byte of
> such constraints, while in HEVC there are more bytes of such
> constraints. I think what you were thinking of is some optimization.
> Considering the urgent situation that the RFC is urgent needed by the
> relevant industries for HEVC deployment, I personally would prefer to
> go ahead without trying to make more optimization here, but let it be
> similar as in RFC 6184 and RFC 6190. However, if the group thinks
> that we should do more optimization here, then we can try it, but
> this needs careful study and investigation.

No, I am not talking about optimizations here. I don't even see how they
work at all. Especially the profile compatibility where they potential
usage is being prevented by other payload type rules. Thus, I think we
need to dig into this, or simply remove it from the parameters as it
provides no benefit. Or please explain how it can provide any benefit.

I think the interop flags, at least the current set will work as
receiver primarily and even sender constraints for bi-directional media
blocks. If you have different set of constraints that you want to
express you will need to offer and answer with multiple payload types,
each expressing the different sets.

But, the basic issue here has to do with what type of negotiation if any
and what applicability the constraint have. I assume on the basic level
anyone using the PT will have to have their stream fulfil the signalled
constraints. Thus, they are receiver only. By, mandating them to the "C"
category, i.e. copy then they becomes sender constraints. Thus, no
possibility to negotiate them. Just express them. I think I am fine with
this, but the issue is that you don't know what future interop
constraints that will be defined and what applicability it will have. If
it is something that you actually want to negotiate, then you start
exploding the PT combinations.

See inline for response to the other questions.

Cheers

Magnus

>>> w/o recv-sub-layer-id --+  |  | answer: sendrecv,
>>> recv-sub-layer-id --+ |  |  | sendrecv w/o recv-sub-layer-id --+
>>> |  |  |  | |  |  | |  | profile-space                      C  X
>>> C  X  P profile-id C X  C  X  P tier-flag
>>> C  X  C  X  P level-id                           C  X  C  X  P
>>> 
>>> So these parameters are receiver capabilities, and they needs to
>>> be symmetrically used in the signalling, however the actually
>>> used values in the transmission direction from an O/A agent only
>>> needs to be within the receivers provided parameters. Thus, they
>>> are not a "configuration" for the sending direction.
>>> 
>>>  what is to be sent and what is to be received for sendrecv and
>>> must be used symmetrically, and only the latter for recvonly.
>> 
> [YK]: Could you please clarify what does it mean by "not combined",
> and by "to use some other property identifier"?

With combined, I meant the arrangement in H.264 with profile-level-id,
i.e. the profile and level was combined parameters.

With other property identifier, I meant replacing the "C" with some
other character expressing the property of this being allowed to downgrade.

>>> [YK]: When recv-sub-layer-id is included in an SDP answer, it
>>> means that the answer chooses one of the multiple operation
>>> points corresponding to the offered or declared sub-layers in the
>> with the same values as in the offer it responds to. Otherwise the
>> PT matching rule fails.
>> 
>>  considering i all the other cases that information will be
>> explicit here. Thus, I think the X's are plain wrong. Or I am still
>> SDP includes a VPS that provides two operation points, 720p@15fps
>> and 720p@30fps, corresponding to recv-sub-layer-id 0 and 1,
>> respectively. The profile, tier, level, and so on for each
>> operation point are signalled by the VPS. If recv-sub-layer-id=0 is
>> included in an SDP answer, then that means the answerer chooses to
>> operate at 720p@15fps. In this case, the profile, tier, level, and
>> so on for the chosen operation point does not need to be explicitly
>> sent again in the SDP, as it is clear what they are (they are
>> signalled in the VPS in the SDP offer). So there is no ambiguity
>> here. Are you suggesting that in some use cases even there is no
>> ambiguity the configuration information must always be explicitly
>> signalled (i.e. values that are derivable)? If yes, could you
>> please explain in what use cases and why. Note that we had the same
>> mechanism specified for SVC in RFC 6190. Also, on PT use, we have
>> allowed the use of a different PT value in an SDP answer than the
>> matching PT value in the offer since at least RFC 3984 as long as
>> the PT matching is clear.
> 
> Offer/Answer. The reason is that you substitute the answering
> explicitly declaring its receiving capabilities with a reference to a
> particular instantiation of a video sequence parameter set that falls
> within the receiver's capabilities. Thus, the offerer does not know
> the receiver's actually capabilities if it needs to reconfigure the
> encoder, or in case the RTP payload type is going to be used by
> multiple media streams and possibly different encoder instances.
> 
> signalling path that it keeps track and state for what was in the
> offer to know what the answer means.
> 
> situation differently, and instead declare its capabilities as
> normally in answer. The recv-sub-layer-id parameter is an optional
> parameter and it is also stream specific, while the profile, level,
> etc are generically applicable to the RTP payload type do applies on
> RTP session level.
> 
> selection mechanism, but does not replace the capability declaration
> the regular parameters perform.
> 

Great!

>>> w/o recv-sub-layer-id --+  |  | answer: sendrecv,
>>> recv-sub-layer-id --+ |  |  | sendrecv w/o recv-sub-layer-id --+
>>> |  |  |  | |  |  | |  | interop-constraints                C  X
>>> C  X  P profile-compatibility-indicator    C  X  C  X  P
>>> 
>>> receiver capability declarations, so I don't understand why the
>>> Cs are C and not P.
>>> 
>>> [YK]: As explained earlier, these are also parts of the media
>>> type configuration, same as profile, tier, and level, and their
>>> usage are the same as those too.
>> 
>> First of all these parameters are not included as the ones that
>> have to be symmetrically provided. I also think they could be SDP
>> agent specifically given. But, that can be discussed. But,
>> according to the current text, I fail to see that these needs to be
>> handled as the profile-id, level-id, etc.
>> 
>> [YK]: Maybe you were initially reading version -01 and did not
>> check version -02 on this part. In version -02, these parameters
>> are included in the list of configuration parameters that are
>> specified to be used symmetrically.
> 
> This issue is related to the interpretation of
> "interop-constraints", and "profile-compatibility-indicator". It is
> not clear in which direction they should be interpreted in the
> various cases. Are they constraints on what the receiver/decoder
> accept or constraints the encoder promise to fulfil or both?
> 
> profile-compatibility-indicator indicating Main and Baseline. This
> means the offerer prefers to do main but knows of baseline. Answerer
> is baseline only. Thus it detects a profile not supported, but
> compatibility indicates support. However, due to the other payload
> type rules, the answerer must reject PT=97 as it can't answer with
> 97=Baseline. Nor can it reject 97 and and 98 which is baseline, and
> only use 97 in answerer to offerer direction. And only receive using
> 98. Especially not if 97 was the only PT offered, then you have no
> compatibility.
> 
> Instead, to my understanding to achieve receiver constraints the
> offerer must include one PT per profile it supports, the
> If this is a sender constraint, then it actually tells something
> about the sender's capability of providing something that matches
> more profiles than what it is willing to receive. However, the
> answerer if it doesn't support the any of the PTs and their
> profile-id values then it anyway have to refuse the PT, even if it
> could receive the stream from the offerer. Instead, also here the
> offerer needs to indicate all profiles it can encoded for and offer
> them. Which implies also decoding capability. In addition, if this is
> a sender constraint, using C as classification (i.e. must copy) in
> answer, if the encoder in the answerer supports the payload types
> profile-id but, can't ensure or know about all the
> profile-compatibility flags, then it has to refuse the payload type
> despite it supporting it.
> 
> To my understanding such an interpretation only forces the case of
> more rejections, if the answerer doesn't have an equally capable
> encoder and decoder as the offerer and can match both sets of
> constraints. In addition the compatibility has not provided any
> benefit for the two O/A agents.
> 
> If I haven't I think these parameters usage is seriously in question.
> If one would like to use it then the must not alter constraint for
> profile becomes a serious issue. If one would remove that then I see
> that using "profile-compatibility-indicator" could be possibly
> beneficial. But, I haven't done a full out design of how to resolve
> the PT matching and deal with asymmetric usage within the session.
> 
> [YK]: They are meant for Receiver and Sender constraints. What we
> have now is similar to what we have in RFC 6184 and RFC 6190, with
> the essential difference been that in AVC there is only one byte of
> such constraints, while in HEVC there are more bytes of such
> constraints. I think what you were thinking of is some optimization.
> Considering the urgent situation that the RFC is urgent needed by the
> relevant industries for HEVC deployment, I personally would prefer to
> go ahead without trying to make more optimization here, but let it be
> similar as in RFC 6184 and RFC 6190. However, if the group thinks
> that we should do more optimization here, then we can try it, but
> this needs careful study and investigation.
> 
> BR, YK
> 
> 
> 

-- 

Magnus Westerlund

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Mon, 28 Apr 2014 22:03:44 +0000

Thanks for the clarification on the first point.

Now again on the third point about profile-compatibility-indicator and interop-constraints.

profile-compatibility-indicator tells the list of profiles the bitstream conforms to. The profile identified by profile-id would also be indicated by profile-compatibility-indicator. The information that is covered by profile-id but not by profile-compatibility-indicator is as follows (note 6 in subclause 7.4.4 of the HEVC spec): NOTE 6 - When the coded video sequence conforms to multiple profiles, general_profile_idc should indicate the profile that provides the preferred decoded result or the preferred bitstream identification, as determined by the encoder (in a manner not specified in this Specification).

Therefore, the use of profile-id and profile-compatibility-indicator together here in this payload format spec is similar as the use in RFC 6184 of profile_idc and profile-iop (the first two bytes of profile-level-id, except the particular and weird indication of level 1b using bit 4 of profile-iop).

What we could do to optimize the O/A herein is to allow for the change of the values of profile-id and profile-compatibility-indicator as long as they represent the same set of profiles. But I am not sure whether or how many PT values could be saved in practice. 

interop-constraints is something new in HEVC compared to AVC. Example constraints here include 1) whether the video is a frame-packed 3D video; and 2) whether the coded pictures are coded frames only or there are interlaced stuff involved. Herein I think it is preferable to require both sides to maintain the values, because if an offerer offers to communicate with clean progressive video without frame packing, it also expects the other side to keep the restrictions and do the same.

BR, YK

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Sunday, April 27, 2014 11:51 PM
To: Wang, Ye-Kui; [email-masked]; [email-masked]
Subject: Re: [payload] Comments on draft-ietf-payload-rtp-h265-02

On 2014-04-25 19:56, Wang, Ye-Kui wrote:
> [YK]: They are meant for Receiver and Sender constraints. What we have 
> now is similar to what we have in RFC 6184 and RFC 6190, with the 
> essential difference been that in AVC there is only one byte of such 
> constraints, while in HEVC there are more bytes of such constraints. I 
> think what you were thinking of is some optimization.
> similar as in RFC 6184 and RFC 6190. However, if the group thinks that 
> we should do more optimization here, then we can try it, but this 
> needs careful study and investigation.

No, I am not talking about optimizations here. I don't even see how they work at all. Especially the profile compatibility where they potential usage is being prevented by other payload type rules. Thus, I think we need to dig into this, or simply remove it from the parameters as it provides no benefit. Or please explain how it can provide any benefit.

I think the interop flags, at least the current set will work as receiver primarily and even sender constraints for bi-directional media blocks. If you have different set of constraints that you want to express you will need to offer and answer with multiple payload types, each expressing the different sets.

But, the basic issue here has to do with what type of negotiation if any and what applicability the constraint have. I assume on the basic level anyone using the PT will have to have their stream fulfil the signalled constraints. Thus, they are receiver only. By, mandating them to the "C"
category, i.e. copy then they becomes sender constraints. Thus, no possibility to negotiate them. Just express them. I think I am fine with this, but the issue is that you don't know what future interop constraints that will be defined and what applicability it will have. If it is something that you actually want to negotiate, then you start exploding the PT combinations.

Cheers

Magnus

>>> |  |  | sendrecv w/o recv-sub-layer-id --+

With combined, I meant the arrangement in H.264 with profile-level-id, i.e. the profile and level was combined parameters.

With other property identifier, I meant replacing the "C" with some other character expressing the property of this being allowed to downgrade.

>>> information of profile, level etc. in the SDP, thus there are marked 
>>> with X (MUST NOT be present).
>> 

Great!

>>> |  |  | sendrecv w/o recv-sub-layer-id --+
> profiles than what it is willing to receive. However, the answerer if 
> it doesn't support the any of the PTs and their profile-id values then 
> it anyway have to refuse the PT, even if it could receive the stream 
> from the offerer. Instead, also here the offerer needs to indicate all 
> profiles it can encoded for and offer them. Which implies also 
> decoding capability. In addition, if this is a sender constraint, 
> using C as classification (i.e. must copy) in answer, if the encoder 
> in the answerer supports the payload types profile-id but, can't 
> ensure or know about all the profile-compatibility flags, then it has 
> to refuse the payload type despite it supporting it.
> 
> [YK]: They are meant for Receiver and Sender constraints. What we have 
> now is similar to what we have in RFC 6184 and RFC 6190, with the 
> essential difference been that in AVC there is only one byte of such 
> constraints, while in HEVC there are more bytes of such constraints. I 
> think what you were thinking of is some optimization.
> similar as in RFC 6184 and RFC 6190. However, if the group thinks that 
> we should do more optimization here, then we can try it, but this 
> needs careful study and investigation.
> 
> BR, YK
> 
> 
> 

-- 

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 30 Apr 2014 15:51:34 +0200

On 2014-04-29 00:03, Wang, Ye-Kui wrote:
> profile-compatibility-indicator tells the list of profiles the
> bitstream conforms to. The profile identified by profile-id would
> also be indicated by profile-compatibility-indicator. The information
> that is covered by profile-id but not by
> profile-compatibility-indicator is as follows (note 6 in subclause
> 7.4.4 of the HEVC spec): NOTE 6 - When the coded video sequence
> conforms to multiple profiles, general_profile_idc should indicate
> the profile that provides the preferred decoded result or the
> preferred bitstream identification, as determined by the encoder (in
> a manner not specified in this Specification).

Yes, I do understand the purpose of the indicator in the relation to a
particular bit-stream. However, this benefit is not possible to utilize
at all in SDP O/A with the rules you specify. As my initial post in this
tried to analyse there are some rule changes required to have it provide
any benefit. I get the impression these values have been thrown into the
payload format signalling without really considering the implications
and limitations that the SDP O/A results in.

> 
> Therefore, the use of profile-id and profile-compatibility-indicator
> together here in this payload format spec is similar as the use in
> RFC 6184 of profile_idc and profile-iop (the first two bytes of
> profile-level-id, except the particular and weird indication of level
> 1b using bit 4 of profile-iop).

They may be similar, but that doesn't mean that their usage was the most
appropriate or best possible before. With a new standard one has the
chance to revisit this. I do understand the pressure to finish this up
as quickly as possible. But, I think you should take the time to
consider how this is used.

> 
> What we could do to optimize the O/A herein is to allow for the
> change of the values of profile-id and
> profile-compatibility-indicator as long as they represent the same
> set of profiles. But I am not sure whether or how many PT values
> could be saved in practice.

If there is nothing to save, then I think you should question if you
gain anything from including the profile-compatibility-indicator as part
of the PT signalling at all?

I see that it can potentially be used to indicate which additional
profiles that a encoder can produce. For that to be useful the answer
needs to consider if it can add new PTs for it. Or if it is only use is
to allow a richer second Offer/Answer round.

The other aspect of this is if it is at all reasonable to expect the
peer encoder to be able to produce the same compatibility pattern, while
the main constraint is to support the actual supported
profile+level+constraints. I question that aspect of using it as
receiver constraint for a particular payload type.

> 
> interop-constraints is something new in HEVC compared to AVC. Example
> constraints here include 1) whether the video is a frame-packed 3D
> video; and 2) whether the coded pictures are coded frames only or
> there are interlaced stuff involved. Herein I think it is preferable
> to require both sides to maintain the values, because if an offerer
> offers to communicate with clean progressive video without frame
> packing, it also expects the other side to keep the restrictions and
> do the same.

Yes, I understand that some of the current parameters, you really want
the answering party to uphold them. One important question is if there
are clear and working rules for unknown constraints. How does the
answerer handle unknown constraints? Here I think a first step is to
explain how unknown value are to be dealt with.

The other aspect of maintaining the interop-constraints values and
really not have any negotiation and instead use different PTs if one
like to offer different values, is the potential explosion of
combinations in the future when more constraints are added.

Cheers

Magnus Westerlund

---

From: Yago Sanchez <[email-masked]>
Date: Wed, 30 Apr 2014 16:40:35 +0200

Hi Magnus,

I think there is a misunderstanding with the intention of using the profile-compatibility-indicator. It is not about indicating support for additional profiles. The profile-compatibility-indicator sets additional constraints to the profile indicated in the profile-id. That is why Ye-Kui was pointing that both the profile-id and the profile-compatibility-indicator are used to express the same as  the profile_idc in RFC 6184. This means that if the offerer indicates profile-id equal to A and includes profile B in the profile-compatibility-indicator, the offerer would indicate capability of encoding/decoding a stream that only contains the tools that are in the intersection of both profiles. The offerer promise would not even be that it is able to decode the full A profile, unless indicated in another PT.

My understanding was that this would be useful for those cases where two endpoints support different profiles but are aware of the intersection with other profiles, as in the case expressed above. Thus, there could express more PTs with the profile they fully support and with subsets of it result of intersections with other profiles.

Therefore, I still think that including profile-compatibility-indicator may be useful. Would this be reasonable or is there anything I am missing from the discussion below?

Best regards,
Yago

On 30 Apr 2014, at 15:51, Magnus Westerlund <[email-masked]> wrote:

> _______________________________________________
> payload mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/payload