Subject: [rtcweb] Comments on draft-ietf-rtcweb-jsep-10

From: Magnus Westerlund <[email-masked]>
Date: Tue, 16 Jun 2015 14:13:15 +0200

Hi,

I have reviewed the changes between the -09 and -10 versions and have 
some comments.

A. Section 3.5:

3.5.  RTP CNAME Semantics

    RTP CNAME values provide a canonical name for the RTP endpoint,
    allowing other RTP endpoints to determine which RTP streams are using
    the same clock and thus which clock sources can be used that to
    synchronize media playout.

    Any MediaStreamTracks which have different clock sources MUST have
    different CNAMEs [TODO: need a reference for this.]  Any
    MediaStreamTracks which are in different PeerConnection objects MUST
    have different CNAMEs; this prevents peers from linking calls from
    multiple remote PeerConnections based on the CNAME.  For simplicity,
    MediaStreamTracks in the same PeerConnection which have the same
    clock source SHOULD have the same CNAME.

The main problem is that this is out-of-synch (Pardon the pun) with what 
RTP-usage specifies. See Section 4.9:

I think the relevant is the following:

    Taking the discussion in Section 11 into account, a WebRTC Endpoint
    MUST NOT use more than one RTCP CNAME in the RTP sessions belonging
    to single RTCPeerConnection (that is, an RTCPeerConnection forms a
    synchronisation context).  RTP middleboxes MAY generate RTP packet
    streams associated with more than one RTCP CNAME, to allow them to
    avoid having to resynchronize media from multiple different endpoints
    part of a multi-party RTP session.

So lets look at the normative statements in JSEP:

   "Any MediaStreamTracks which have different clock sources MUST have
    different CNAMEs [TODO: need a reference for this.]"

This is not allowed by an WebRTC endpoint, which must use only a single 
CNAME. Although a RTP middlebox can forward multiple CNAMEs, the WebRTC 
endpoint is instead required to perform re-synchronization into the used 
clock based of the other streams.

   "Any
    MediaStreamTracks which are in different PeerConnection objects MUST
    have different CNAMEs; this prevents peers from linking calls from
    multiple remote PeerConnections based on the CNAME."

This one is correct, but I don't quite understand the "this prevents 
peers from linking calls from multiple remote PeerConnections based on 
the CNAME." part. First of all what is a "call". Secondly, what type of 
linking are you considering here?

   "For simplicity,
    MediaStreamTracks in the same PeerConnection which have the same
    clock source SHOULD have the same CNAME."

The SHOULD needs to be a MUST.

B. Section 3.6.1:

    Otherwise, an "a=imageattr" attribute is created with "recv"
    direction, and the resulting resolution space formed by intersecting
    the decoder limits and constraints is used to specify its minimum and
    maximum x= and y= values.  If the intersection is the null set, i.e.,
    there are no resolutions that are permitted by both the decoder and
    the mandatory constraints, this SHOULD be represented by x=0 and y=0
    values.

    The rules here express a single set of preferences, and therefore,
    the "a=imageattr" q= value is not important.  It SHOULD be set to
    1.0.

What about end-points that can produce different aspect ratios on its 
video. Such a system is likely to have different constraints on its min 
and max resolution in X and Y for 1:1, 4:3, 16:9 and 1:2.35 aspect 
ratios. It is unclear if that is allowed, and even recommend practice. 
And in this case there might be a usage for the q value.

C. Section 3.6.1:

    As an example, consider a system with a HD-capable, multiformat video
    decoder, where the application has constrained the received track to
    at most 360p.  In this case, the implemention would generate this
    attribute:

    a=imageattr:* recv [x=[16:640],y=[16:360],q=1.0]

Maybe this example should discuss the minimal aspect of this. What it 
really is expressing in that it can handle any resolution between 16x16 
pixels up to 640x360 as the longest in respective side. No limitations 
to any picture aspect ratio.

D. Section 3.6.1:

What about usage of "par" in the sets. I actually expect most have 
limitations to perform a scaling that maintain the PAR, not performing 
cropping or scaling that warps the picture.

E. Section 3.6:

I am missing a discussion on how video orientation affects what one 
states. Taking the above example with a maximum resolution of 640x360, 
which gives a landscape 16:9 PAR. If one rotates this, then it is not 
clear if one is allowed to have 360x640 or are limited to 203x360. Or 
should one actually state the different orientations maximum resolution 
and consider that this is simply different PAR, as 16:9 (Landscape) is 
9:16 (Portrait) two different picture aspect ratios.

F. Section 4.1.1:

    Regardless of policy, the
    application will always try to negotiate BUNDLE onto a single
    transport, and will offer a single BUNDLE group across all media
    sections.

I think this sentence may be a bit misleading without additional 
sentence(s) to explain that the actual number of established transport 
streams will depend on the peer's response to the attempt to bundle.

G. Section 5.2.1:

This section contains the following:

    Each m= section MUST include the following attribute lines:

    o  If this m= section is for video media, an "a=imageattr" line, as
       specified in Section 3.6.

But, Section 3.6 does not mandate that a=imageattr always is included. 
Thus the text must be modified to allow the selective inclusion of the 
attribute based on what needs to be said.

H. Section 5.2.1:

    o  An "a=ssrc" line, as specified in [RFC5576], Section 4.1,
       indicating the SSRC to be used for sending media, along with the
       mandatory "cname" source attribute, as specified in Section 6.1,
       indicating the CNAME for the source.  The CNAME must be generated
       in accordance with [RFC7022] and Section 3.5.

I think this document shouldn't make normative statement to how CNAMEs 
shall be generated. That last sentence points to RFC7022, but that is 
not the whole story. If you like to point to how CNAMEs are to be 
generated then say that Section 4.9 of RTP-usage is the place to 
determine that.

I. Section 5.3.1:

    The next step is to generate lip sync groups as defined in [RFC5888],
    Section 7.  For each MediaStream with more than one MediaStreamTrack,
    a group of type "LS" MUST be added that contains the mid values for
    each MediaStreamTrack in that MediaStream.  In some cases this may
    result in adding a mid to a given LS group that was not in that LS
    group in the associated offer.  Although this is not allowed by
    [RFC5888], it is allowed when implementing this specification.

So, do I understand it correctly the violation this paragraph is 
discussing is the following from Section 9.2 of RFC5888:

    The
    identification-tags contained in this "a=group" line MUST be the same
    as those received in the offer, or a subset of them (zero
    identification-tags is a valid subset).

So when will these violations occur for a LS group in JSEP. I assume due 
to that offer and answer has different structure of how MediaStreams 
uses the different m= lines. But, isn't this just another case of when 
an answerer will have to do an follow up Offer to be able to add all the 
MediaStreamTracks and their MediaStreams due to different sets on the 
two peers?

I don't really think this specification should override RFC 5888.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Justin Uberti <[email-masked]>
Date: Wed, 17 Jun 2015 20:52:21 -0700

On Tue, Jun 16, 2015 at 5:13 AM, Magnus Westerlund <
[email-masked]> wrote:

> Hi,
>
> I have reviewed the changes between the -09 and -10 versions and have some
> comments.
>

Thanks for the feedback. Comments inline; I will also try to cover any
issues identified here in tomorrow's presentation.

> clock based of the other streams.

TBH, I wasn't sure exactly where it was mandated that different clock
sources translate into different CNAMEs (hence the missing ref). Cullen, do
you know?

If there is no problem with using a single CNAME in this case, I think that
is clearly the simplest thing to do.

> This one is correct, but I don't quite understand the "this prevents peers
> from linking calls from multiple remote PeerConnections based on the
> CNAME." part. First of all what is a "call". Secondly, what type of linking
> are you considering here?
>

"call" may not be the ideal terminology here, but the concern here is an
application which makes a connection to peer A with PC A and then later
makes a connection to peer B with PC B. If CNAME was reused, someone with
knowledge of A and B could see that they received calls from the same
originating endpoint.

> The SHOULD needs to be a MUST.

If we follow the advice from rtp-usage as mentioned above, I agree.

> video. Such a system is likely to have different constraints on its min and
> max resolution in X and Y for 1:1, 4:3, 16:9 and 1:2.35 aspect ratios. It
> is unclear if that is allowed, and even recommend practice. And in this
> case there might be a usage for the q value.
>

Recall that these are "recv" direction attributes, so what can be produced
is not in question here.

I do see a valid point in that a receiver might want to optionally request
a specific aspect ratio on incoming video (perhaps inferred from optional
constraints). That seems like a matter for further study, but does not
contradict any of the existing text.

> pixels up to 640x360 as the longest in respective side. No limitations to
> any picture aspect ratio.
>

I was trying to give a reasonably real-world example. Do you think this is
confusing?

> cropping or scaling that warps the picture.

I agree that scaling should be done at 1:1, but not sure exactly what you
think ought to be specified here.

> which gives a landscape 16:9 PAR. If one rotates this, then it is not clear
> if one is allowed to have 360x640 or are limited to 203x360. Or should one
> actually state the different orientations maximum resolution and consider
> that this is simply different PAR, as 16:9 (Landscape) is 9:16 (Portrait)
> two different picture aspect ratios.
>

This is a good point. I would like to assume the limit applies to the
horizontal orientation and the video will be sent in that format, with CVO
used to handle rotation (since we are mandating support for CVO).

> streams will depend on the peer's response to the attempt to bundle.

OK, I am fine with including a sentence to mention the "else" case.

> attribute based on what needs to be said.

I tried to spell this out in 3.6, but if you think it needs to be
reiterated here as well that is fine by me.

> shall be generated. That last sentence points to RFC7022, but that is not
> the whole story. If you like to point to how CNAMEs are to be generated
> then say that Section 4.9 of RTP-usage is the place to determine that.
>

I just read over 4.9, and I agree. In fact, Section 3.5 in JSEP should
probably mostly be a pointer to Section 4.9 in rtp-usage.

> to that offer and answer has different structure of how MediaStreams uses
> the different m= lines. But, isn't this just another case of when an
> answerer will have to do an follow up Offer to be able to add all the
> MediaStreamTracks and their MediaStreams due to different sets on the two
> peers?
>

No. Consider this trivial case: A offers two streams (audio and video),
unsynced. B replies with a single stream (with audio and video), to be
synced.

A will have two (or zero) LS groups, but B wants to have a single LS group
with two mid entries.

This is not a completely hypothetical case, as A could be a screenshare
endpoint.

> I don't really think this specification should override RFC 5888.

I don't see any other way to make this work.

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 18 Jun 2015 16:04:28 +0200

Hi,

Please see inline. I will add these to the github issue tracker.

Justin Uberti skrev den 2015-06-18 05:52:
>
>
> On Tue, Jun 16, 2015 at 5:13 AM, Magnus Westerlund
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     Hi,
>
>     I have reviewed the changes between the -09 and -10 versions and
>     have some comments.
>
>
>     The main problem is that this is out-of-synch (Pardon the pun) with
>     what RTP-usage specifies. See Section 4.9:
>
>     This is not allowed by an WebRTC endpoint, which must use only a
>     single CNAME. Although a RTP middlebox can forward multiple CNAMEs,
>     the WebRTC endpoint is instead required to perform
>     re-synchronization into the used clock based of the other streams.
>
>
> sources translate into different CNAMEs (hence the missing ref). Cullen,
> do you know?

Depends of what one means with clock sources. However, in RTP one need 
to be able to provide the RTP Timestamp value of media source in 
relation to the endpoints reference clock (in NTP format). All SSRCs 
with the same CNAME needs to have the same reference clock otherwise 
they are not possible to synchronize.

>
> If there is no problem with using a single CNAME in this case, I think
> that is clearly the simplest thing to do.

Clearly re-synchronization is not trivial, but there are complexities 
with having WebRTC endpoint sending multiple synchronization contexts 
also. But, I expect that at some point in the future one might want to 
change this behaviour.

>     peers from linking calls from multiple remote PeerConnections based
>     on the CNAME." part. First of all what is a "call". Secondly, what
>     type of linking are you considering here?
>
>
> makes a connection to peer B with PC B. If CNAME was reused, someone
> with knowledge of A and B could see that they received calls from the
> same originating endpoint.

I think my main problem was the usage of call. In grouping taxonomy I 
think you are talking about Multimedia Sessions. But, can't you simply 
talk about PeerConnections. The grouping taxonomies larger term is 
communication session, which would be potentially multiple multimedia 
sessions (PCs) with a common context.

The tracking issue you are talking about is already covered in Section 
4.9 of RTP usage:

Accordingly, a
    WebRTC Endpoint MUST generate a new, unique, short-term persistent
    RTCP CNAME for each RTCPeerConnection, following [RFC7022], with a
    single exception; if explicitly requested at creation an
    RTCPeerConnection MAY use the same CNAME as as an existing
    RTCPeerConnection within their common same-origin context.

> If we follow the advice from rtp-usage as mentioned above, I agree.

I think you have no choice! JSEP can't override other documents at will.

>     What about end-points that can produce different aspect ratios on
>     its video. Such a system is likely to have different constraints on
>     its min and max resolution in X and Y for 1:1, 4:3, 16:9 and 1:2.35
>     aspect ratios. It is unclear if that is allowed, and even recommend
>     practice. And in this case there might be a usage for the q value.
>
>
> Recall that these are "recv" direction attributes, so what can be
> produced is not in question here.

But, a=imageattr can express send direction limiations also. That is not 
discussed currently. And also for receiving an receiver can have 
different capabilites depending on picture aspect rate.

>
> I do see a valid point in that a receiver might want to optionally
> request a specific aspect ratio on incoming video (perhaps inferred from
> optional constraints). That seems like a matter for further study, but
> does not contradict any of the existing text.

That is one possibility, the other is that one actually have limiations 
that vary with what PAR is to be used.

>     Maybe this example should discuss the minimal aspect of this. What
>     it really is expressing in that it can handle any resolution between
>     16x16 pixels up to 640x360 as the longest in respective side. No
>     limitations to any picture aspect ratio.
>
>
> I was trying to give a reasonably real-world example. Do you think this
> is confusing?

No, it is not confusing, but I think it could be more explicit and also 
mention what else it says, not only the maximum, but also a minimal of 
16x16 pixels with 1 pixel steps in both directions.

>     limitations to perform a scaling that maintain the PAR, not
>     performing cropping or scaling that warps the picture.
>
>
> I agree that scaling should be done at 1:1, but not sure exactly what
> you think ought to be specified here.

I think the important to make clear is that limitations may be PAR 
dependent.

>     states. Taking the above example with a maximum resolution of
>     640x360, which gives a landscape 16:9 PAR. If one rotates this, then
>     it is not clear if one is allowed to have 360x640 or are limited to
>     203x360. Or should one actually state the different orientations
>     maximum resolution and consider that this is simply different PAR,
>     as 16:9 (Landscape) is 9:16 (Portrait) two different picture aspect
>     ratios.
>
>
> horizontal orientation and the video will be sent in that format, with
> CVO used to handle rotation (since we are mandating support for CVO).

So you are saying is that if one have y=[16:640], y=[16:360] then one 
should be capable of handling a 90 degrees rotated video that has to 
display in portrait.

It needs to be specified, lets see if you get protests on that 
interpretation.

>     sentence(s) to explain that the actual number of established
>     transport streams will depend on the peer's response to the attempt
>     to bundle.
>
>
>     But, Section 3.6 does not mandate that a=imageattr always is
>     included. Thus the text must be modified to allow the selective
>     inclusion of the attribute based on what needs to be said.
>
>

The issue is that the structure results in contradiciting the Section 
3.6 thus I think one need to consider if one needs to change the 
formulation of the MUST statement or the bullet to avoid that.

>     I think this document shouldn't make normative statement to how
>     CNAMEs shall be generated. That last sentence points to RFC7022, but
>     that is not the whole story. If you like to point to how CNAMEs are
>     to be generated then say that Section 4.9 of RTP-usage is the place
>     to determine that.
>
>

Please do.

>     So when will these violations occur for a LS group in JSEP. I assume
>     due to that offer and answer has different structure of how
>     MediaStreams uses the different m= lines. But, isn't this just
>     another case of when an answerer will have to do an follow up Offer
>     to be able to add all the MediaStreamTracks and their MediaStreams
>     due to different sets on the two peers?
>
>
> A will have two (or zero) LS groups, but B wants to have a single LS
> group with two mid entries.
>

Thanks for providing cases when it can occur.

> I don't see any other way to make this work.

There is one obvious solution, the answering part needs to initiate an 
offer by themselves. We have several such cases already, for example 
answer adding more media streams than the offer. And you have previously 
argued that a negotiation isn't that expensive, so I see that is the 
reasonable way of handling this limiation.

I think we shouldn't override RFC5888 due to implications that has on 
WebRTC endpoint to WebRTC compatible endpoints. It can make them throw 
up if they do support LS groups.

Cheers

Magnus Westerlund