Subject: [rtcweb] MediaStream Label and CNAME

From: Magnus Westerlund <[email-masked]>
Date: Tue, 13 Sep 2011 10:43:23 +0200

WG,
(As an individual contributor)

There has been some discussion as result of the presentation of
terminology in the RTCWEB Interim meeting last Thursday. The biggest
question was why CNAME can't map to MediaStream label. Below we
clarify why we think CNAME and label are separate entities.

One part in this reasoning has to do with the current definition of
’media resource’
(<http://dev.w3.org/html5/spec/Overview.html#media-resource>) and media
elements of html5. The ‘media resource’ could be a file, or, more
relevant to this discussion, a MediaStream. In that usage only a single
video track can be played simultaneously and in sync with one or more
audio tracks.

Thus unless we modify an existing semantics the only way of playing
multiple video tracks in sync with one or more audio tracks is to have
multiple MediaStream objects.

We see the need for supporting multiple synchronized video tracks. We
have at least two basic use cases.

1) An endpoint has two or more cameras / video grabbers that a web
application wants to use. The user will select different cameras for use
for different aspects in the application. Naturally the use would expect
audio and video to be synced regardless of camera being viewed.

2) In the use case for centralized mixers one possible video conference
RTP mixer usage is an application that takes a number of participants
audio and mixes that into a single audio stream. The video streams are
selected from among the available and mapped to a number of SSRCs that
is the mixers. All these MediaStreams are mapped to the mixers
synchronization context represented by a CNAME owned by the mixer.

With the current MediaStream semantics this requires multiple objects
with different combinations of media tracks to enable simultaneous
playback in the browser in a synchronized fashion.

We also have some more advanced use cases where having multiple
MediaStreams with different but overlapping tracks makes sense.

The first is a mesh application with four peers (A-D). If the
application in A wants to send Audio plus Video1 to B, and Audio and
Video2 to C and Audio plus both Videos to D then that would be
different MediaSteam objects unless we change the semantics.

We don't believe in modifying the MediaStream semantics unless really
necessary. One reason is that we desire an API and behavior that doesn't
make the synchronization between streams dependent on how you program
things. That can easily occur if the MediaStream object is the sole
method for achieving playback synchronization. Another is that
MediaStream ties in nicely with the existing html5 media elements. We
also see the MediaStream as good choice of representation for the
JavaScript developer as it represents resources that commonly should be
played together.

This argument about programming things in different ways may sound
abstract. But in reality it is a simple as synchronization should be
possible independent if a programmer calls getUserMedia multiple times
rather than trying to clone and disable tracks in a MediaStream object
containing all resources of interest.

In addition, not supporting synchronization between multiple
MediaStreams from a single end-point makes it difficult when the number
of tracks a single end-point wishes to transmit changes. This does occur
for two reasons. One is that one in fact have an application where one
desire to have the functionality to add additional cameras or media
grabber devices. The second one is in the centralized conference cases
where the central node may make changes to how it mixes or switches
between media resources.

Assuming no structural changes we do see a need for having all
media tracks originating from one browser instance to have the same
CNAME to enable sync across tracks belonging to different MediaStreams.
Thus resulting in the label for a MediaStream being different from the
CNAME. The CNAME is in principle a hidden property (at the API level) of
a track enabling playback synchronization across multiple MediaStreams
having tracks with the same CNAME.

In addition we do see the need for having an optimization where a track
being part of multiple MediaStreams are only sent once in PeerConnection
despite belonging to multiple MediaStream associated with that
PeerConnection.

We do understand that there are other ways of meeting the goals that
we have around tracks and their synchronization and application handles.

- A track must be possible to synchronize with any other from the same
sync context

- The sync identifier must be common over multiple PeerConnection
objects within an browser instant.

- The programmer needs a logical identifier for media resources that can
be transferred between the peers.

- Synchronization should not be dependent on how the implementor
actually write the code. If it is possible it should just happen.

Given our desire to avoid redefining the semantics of MediaStream we saw
that keeping the MediaStream label and the CNAME as separate identifiers
is required.

We have also considered the issue of how to signal the MediaStream
label between the two ends of a PeerConnection.

Ericsson's suggestion for how to realize this to use the a=ssrc
attribute as defined in RFC 5576 carried in the SDP if SDP offer/answer
semantics will be used. Otherwise we assume similar information
structures can be created.

A MediaStream object will have a label, e.g.
"61ca6552-968a-435d-88d9-a4727f2ed515". The MediaStream will contain
tracks, each track is mapped to an SSRC in a particular RTP session. In
the SDP the SSRC part of the MediaStream will in its list of a=ssrc
attributes contain one that reads

"a=ssrc:345678123 mslabel=61ca6552-968a-435d-88d9-a4727f2ed515"

In the case a particular track is part of multiple MediaStream objects
the SSRC will have multiple mslabel values, one for each MediaStream
label. As the CNAME is required to be included when using the a=ssrc
attribute the receiving end-point will also a priori know how these
tracks relate to synchronization contexts.

This signaling may be a bit of a problem for legacy applications where
a signaling gateway commonly will not know the legacy client's SSRC
values prior to it actually sending MediaStreams. To support this use
case we do suggest that any received SSRC for which explicit MediaStream
indication hasn't happened is automatically assigned its own MediaStream
with local generated labels. The receiving web application can combine
multiple MediaStreams to a single MediaStream if necessary. As we
propose the CNAME as a hidden property of the underlying track playback
sync will still be possible.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 13 Sep 2011 16:32:03 +0200

> multiple MediaStream objects.
Sorry, I think this can be handled with the existing API without any issue.
Apologies to those who know JavaScript better than me, but this is a handler
that can handle a new incoming video stream

NewStreamHandler(stream) {
    myMultiVideoStream = stream
    myFirstVideo = new MediaStream(myMultiVideoStream)
    myFirstVideo.videoTracks.select(1)
    oneVideoObject.setSource(myFirstVideo.getUrl())
    mySecondVideo = new MediaStream(myMultiVideoStream)
    mySecondVideo.videoTracks.select(2)
    anotherVideoObject.setSource(mySecondVideo.getUrl())
}

I think the API can be improved, but the improvements are unlikely to 
lose this functionality.
(Remember that the API objects are the control surfaces on the stream - 
the video data will likely go straight from the incoming buffer, through 
the codec for decoding, and blitted onto the screen; the "copying" from 
one MediaStream to another MediaStream is purely conceptual.)

I don't think it makes sense to discuss the other points before getting 
this one put to rest.

                  Harald

---

From: "Olle E. Johansson" <[email-masked]>
Date: Tue, 13 Sep 2011 16:38:29 +0200

13 sep 2011 kl. 16:32 skrev Harald Alvestrand:

> I think the API can be improved, but the improvements are unlikely to lose this functionality.
> (Remember that the API objects are the control surfaces on the stream - the video data will likely go straight from the incoming buffer, through the codec for decoding, and blitted onto the screen; the "copying" from one MediaStream to another MediaStream is purely conceptual.)
> 
> I don't think it makes sense to discuss the other points before getting this one put to rest.
> 
>     
Just to add to the stew:

 - For each stream  (video, text, audio) you have one outbound and at least one inbound
 - If we are going to support SDP and SIP, you will end up with multiple inbound streams - one at a time or at the same time

Forking may lead to multiple incoming streams for one outbound "call". Early media may lead to one incoming stream, which may be replaced by another endpoint at "answer time".

I think one important question here is if we are going to allow this at all. If we follow the SIP model we just have to. 

If possible, I would like to see a restriction in rtcweb so that we push this complexity to the server, far away from the browser.

/O

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 13 Sep 2011 16:51:20 +0200

>   - For each stream  (video, text, audio) you have one outbound and at least one inbound
Nit: I'm not sure what "stream" refers to in this sentence.

For the "hockey game" use case, you have 2 video media stream tracks 
outbound, presumably from the same PeerConnection, possibly in the same 
MediaStream (they're in sync), and carried over the same RTP session, 
too. (That one was added at least partially to make sure we'd allow 
multiple outbound video streams).
> If possible, I would like to see a restriction in rtcweb so that we push this complexity to the server, far away from the browser.
My preferred place-to-end-up would be that the browser supports media 
streams and media stream tracks being added and removed at any time 
during the call, without attaching special semantics to them, and the 
Javascript being left to sort out whether it's "early", "conference" or 
"other".

---

From: "Olle E. Johansson" <[email-masked]>
Date: Tue, 13 Sep 2011 16:58:18 +0200

> Nit: I'm not sure what "stream" refers to in this sentence.
Sorry - RTP streams. (I'm viewing this with my old SIP eyes...) But yes, that concept will change in rtcweb with multiplexing.
> 
> For the "hockey game" use case, you have 2 video media stream tracks outbound, presumably from the same PeerConnection, possibly in the same MediaStream (they're in sync), and carried over the same RTP session, too. (That one was added at least partially to make sure we'd allow multiple outbound video streams).
> My preferred place-to-end-up would be that the browser supports media streams and media stream tracks being added and removed at any time during the call, without attaching special semantics to them, and the Javascript being left to sort out whether it's "early", "conference" or "other".

Ok, I use my right to change opinion. Let's totally leave the notion of a "phone call" or "video call" behind.

Thanks for the feedback :-)

/O

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 14 Sep 2011 11:53:57 +0200

> I don't think it makes sense to discuss the other points before getting 

Harald,

If I read this correctly, what you are doing is to creating two media
streams based on a first media stream. Thus you have multiple
mediaStream objects that contains tracks that has a common
synchronziation context. So from my point of view you have very well
demonstrated why you need CNAME to be a property associated with the
tracks in the media stream and not be the label for the mediaStream.

Or how do you see the label for the media stream being handled when you
create two mediaStream objects from the incomming "stream" that has one
label?

Cheers

Magnus Westerlund

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 14 Sep 2011 12:21:22 +0200

> tracks in the media stream and not be the label for the mediaStream.
That's a good point.

My thinking when typing this was that stuff that happens inside a 
browser is synchronized "enough" by default, so that we don't need to 
carry the label along in order to keep synchronity between myFirstVideo 
and mySecondVideo. This may or may not be optimistic; input sought.

The audio tracks inside myFirstVideo are declared as synchronized to the 
myFirstVideo video track by virtue of being inside the same MediaStream 
object; this carries over to them being synchronized if myFirstVideo is 
connected to another PeerConnection, if this is allowed (see "recording" 
discussion). (In this case, I think the CNAME should NOT be the same as 
that coming in over the incoming PeerConnection).
> create two mediaStream objects from the incomming "stream" that has one
My original thinking was that a CNAME is created for any MediaStream at 
creation time, so that we're operating with three CNAMEs in the example 
above.

Another possibility is that CNAME is a property of the attachment of a 
MediaStream to a PeerConnection object; this has the advantage that the 
CNAME property is only defined in the cases where we need it, but we 
don't have to carry the baggage of actually generating CNAMEs for 
MediaStreams that are not connected to PeerConnections.

---

From: Matthew Kaufman <[email-masked]>
Date: Wed, 14 Sep 2011 15:16:51 +0200

On 9/13/11 10:43 AM, Magnus Westerlund wrote:
>   ...In the SDP the SSRC part of the MediaStream will in its list of a=ssrc
> the SSRC will have multiple mslabel values...

And so we continue down the path of requiring the Javascript API to emit 
and consume SDP...

Matthew Kaufman

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 14 Sep 2011 15:29:57 +0200

Hi,

See below.

> and mySecondVideo. This may or may not be optimistic; input sought.

I am not certain that would work well. In reality you do need to keep
track of all your media resources against some reference clock. As
offset etc are depending on the context the streams comes from I think
in most cases you do need to track them. You can't rely on them being
sync only because they are local.

My suggestion is that the CNAME never is exposed in the API. It is
something that happens under the hood and only really present when
transmitting the tracks over RTP where each SSRC will expose the CNAME
it is associated with.

> that coming in over the incoming PeerConnection).

Can we please keep the forwarding of media stream separate from just the
basics. I will come back to the forwarding case and my view on how CNAME
and label needs to be handled in these cases.

Lets start with the basic case when you media stream cloning/forking on
the receiver side. A singel mediaStream with a label and one audio and 2
video is received. They are from one synchronization context. You clone
the media stream into a second one where only the second video is
selected for playback and the first MS has the audio and the first video.

If one uses the received CNAME as indicator that they are the same
context and play them back this works fine. But unless the label and
cname is inherited and kept you broken sync. And you have in fact
mandated a less from obvious implementation method to achieve
synchronized playback. But it requires correct implementation on the
sending side.

Also, if the application uses the data channel or a channel over the
webserver to tell each other about what it is doing with the different
tracks, the label either needs to be different for the different clones
or you can't separate which mediaStream object you are referring to.

If we look at this from the senders perspective, the sender must create
a single media stream in an object it can't directly play back locally
for self view without splitting as it contains two video tracks.

And if the application want to distribute the second camera as an
identifiable labeled entity and still be synchronized with the first
camera and the audio I don't understand how to accomplish this with the
current API unless you can have multiple mediaStreams with still allows
for sync.

If relaying of media is allowed I have the following view. I think the
CNAME should be maintained if the media in itself isn't modified. If it
is mixed or changed, or combined in the node, then a new CNAME shall be
created.

When it comes to the MediaStream label it will implicitly be the same.
You receive the MediaStream from one PeerConnection. Then you add it to
another. In this JS instance it is the same MediaStream. You just have
to keep in mind if you refer to operations that should be done on the
transmission of that MS or local operations on it (thus more associated
with what you receive).

> creation time, so that we're operating with three CNAMEs in the example 

I think that is a bad idea. This as it doesn't match the synchronization
context of the actual stream. All tracks in all three MS are in fact
from the same sync context.

It also makes it very easy for a careless implementor to destroy the
information it in fact may want.

> MediaStreams that are not connected to PeerConnections.

My view is that it should be associated with the actual track source
node. So any media coming from a local device in a browser instance will
have the same CNAME as they are related to the same wall clock and
captured in a common time space. As it is unlikely that the implementor
of the webapp actually can determine if any source is coming from
another physical context. And capturing multiple physical contexts on a
common time base isn't a problem as long as all individual tracks that
one wants to capture are on the same time base.

I would also like to point out that due to that an implementor may call
getusermedia more than once in the program, thus creating different
MediaStream objects but for a potential overlapping set of underlying
media sources. Shouldn't these be possible to sync as they in reality
are as they are coming from all local devices?

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 14 Sep 2011 15:55:24 +0200

> And so we continue down the path of requiring the Javascript API to emit 

Well, as there are no other API proposal that exists in the W3C WG
currently I don't know what else I can use in the examples.

But in fact what is needed is a signaling channel between the peers that
can carry a list of RTP Session:SSRC value = label value. I think that
can be applied to a number of different signaling and API alternatives.

Cheers

Magnus Westerlund