Subject: [rtcweb] Call for Consensus on Use Case for Screen/Application/Desktop sharing

From: Magnus Westerlund <[email-masked]>
Date: Mon, 19 Sep 2011 09:02:00 +0200

WG,

There where some discussion in the Interim meeting last week about a
Screen/Application/Desktop sharing support use case. My take away from
the discussion is that this use cases is likely well enough understood
to actually start a consensus call now. However, to us WG chairs it was
clear that the use case in question actually needs to be split into two
parts.

A) Where the RTCWEB enabled browser can use a local application window,
the whole desktop or a Screen as a media source that can be encoded and
transported over the peerConnection for displaying/playback at the peer.

B) Where a remote peer can provide one or more input types such as mouse
and keyboard to control the local system, not only including the
browser, but also other operating system resources. This clearly can
only happen after additional consent, most likely on a per occasion
consent.

My interpretation is that A only allows for application sharing in
conferencing contexts, like in the WEBEX session the Interim meeting was
held with where we shared slides. Where the combination of A and B is
providing for VNC/Remote desktop.

Thus the question to the WG is the following.

1) Do you support or object the inclusion of use case A, B or Both in
our Use case document?

2) Do you have additional comments for or against either of the use cases?

As WG chair

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 19 Sep 2011 09:28:58 +0200

> transported over the peerConnection for displaying/playback at the peer.
I think this is a fairly well defined applicaiton (basically, take input 
from a synthetic source, where the synthetic source may be anything 
available to the browser), the baseline (but not the optimum) could 
simply be encoding the stuff using a default codec, and the access 
procedures shouldn't be much different than for microphone and camera. I 
support its inclusion in the document.

Note: The implementation of such a feature is far from trivial. I would 
not want to require that all browsers implementing RTCWeb support the 
feature.
> only happen after additional consent, most likely on a per occasion
I see this as a  more tricky thing to get right (in most apps, the 
mixing of events from multiple sources depends strongly on both proper 
timing/sequencing and reliable delivery). I would like to not address 
this for now (RTCWeb version 1).
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: "Olle E. Johansson" <[email-masked]>
Date: Mon, 19 Sep 2011 09:32:20 +0200

> 1) Do you support or object the inclusion of use case A, B or Both in
Yes.
> 2) Do you have additional comments for or against either of the use cases?
Nitpick: Case A also requires user consent.

/O
> Magnus Westerlund

---
* Olle E Johansson - [email-masked]
* Cell phone +46 70 593 68 51, Office +46 8 96 40 20, Sweden

---

From: Saúl Ibarra Corretgé <[email-masked]>
Date: Mon, 19 Sep 2011 09:38:06 +0200

> 1) Do you support or object the inclusion of use case A, B or Both in

I'd go for both.

> 2) Do you have additional comments for or against either of the use cases?

As Harald already said, this is far from being an easy task, so I wonder if this should be defined as a standardized use case or a 'mandatory feature'.

--
Saúl Ibarra Corretgé
AG Projects

---

From: "Olle E. Johansson" <[email-masked]>
Date: Mon, 19 Sep 2011 09:50:58 +0200

19 sep 2011 kl. 09:28 skrev Harald Alvestrand:

> I think this is a fairly well defined applicaiton (basically, take input from a synthetic source, where the synthetic source may be anything available to the browser), the baseline (but not the optimum) could simply be encoding the stuff using a default codec, and the access procedures shouldn't be much different than for microphone and camera. I support its inclusion in the document.
> 
> Note: The implementation of such a feature is far from trivial. I would not want to require that all browsers implementing RTCWeb support the feature.
I can agree with that, but I want to explore how this use case would affect RTCweb architecture.

> I see this as a  more tricky thing to get right (in most apps, the mixing of events from multiple sources depends strongly on both proper timing/sequencing and reliable delivery). I would like to not address this for now (RTCWeb version 1).
I think it's a good use case for the data channel. How many such use cases do we have? While use case A is quite often handled as a normal video stream, use case B is likely something like VNC. This is an application that is part of Microsoft Lync as well as the free SIP client Blink today.

/O

---

From: Randell Jesup <[email-masked]>
Date: Mon, 19 Sep 2011 04:01:45 -0400

> 1) Do you support or object the inclusion of use case A, B or Both in

Both.

> 2) Do you have additional comments for or against either of the use cases?
IMHO: Both use-cases involve extra-fun security issues, but both are actually
pretty compelling and useful cases to users.  Everyone I mentioned them to
at the Mozilla All-Hands event "got" the wish for these use-cases pretty
immediately.

VNC-like transfer of screen data is a particularly tough one for the current
permission and access model to enable, but that may change.  A lot of this is
likely to be tied-to or echo the security model and concerns for HTML5 "installed
webapps".

-- 
Randell Jesup
[email-masked]

---

From: "Olle E. Johansson" <[email-masked]>
Date: Mon, 19 Sep 2011 10:11:45 +0200

19 sep 2011 kl. 10:01 skrev Randell Jesup:

>  A lot of this is

Randell,
Do you have any pointers to this security model for further reading?

Thanks,
/O

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 19 Sep 2011 11:23:14 +0200

Olle,

>> 1) Do you support or object the inclusion of use case A, B or Both in

Can you please clarify if Yes, means Both cases?

> Nitpick: Case A also requires user consent.

Fully agree.

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 19 Sep 2011 11:32:43 +0200

As Individual

> 1) Do you support or object the inclusion of use case A, B or Both in

I think A only should be dealt with now.

The reason being that handling a local synthetic source is from this WGs
standardization almost without additional requirements. There is a bit
of extra API demands. There is the extra dimension of consent, where I
think this type needs user consent every time to pick the whole screen
or a particular application instance.

The only real question for us would be if the application output is fine
to encode using the video codecs that are proposed or if the media
encoding uses should be specialized for synthetic content?

As a question on the requirement I would also ask is sound output from a
particular application also should be included? From a requirement point
of view it doesn't seem difficult. Implementation may be far from easy.

Cheers

Magnus Westerlund

---

From: "Olle E. Johansson" <[email-masked]>
Date: Mon, 19 Sep 2011 12:23:06 +0200

> Can you please clarify if Yes, means Both cases?
THis particular yes was for case A, but I agree with both cases as use cases that will affect the architecture. I don't necessarily mean that these apps has to be part of the RTCweb core. 

Monday morning. A bit more poor clarity than other days :-)

>> Nitpick: Case A also requires user consent.
Ok.

/O

---

From: Hadriel Kaplan <[email-masked]>
Date: Mon, 19 Sep 2011 10:39:19 +0000

> 2) Do you have additional comments for or against either of the use cases?

I think scenario A is less worrisome than B.  Scenario A is a lot like webex, obviously, and from a logical perspective if we're going to be able to take a local video file as input to a PeerConnection LocalMediaStream, and the video file could be of me doing stuff on my screen, then this is just a shortcut to taking the video and sending the file at the same time.

Scenario B, on the other hand, is the other half of VNC and my IT dept. would probably freak if a browser could do this, even with user consent.  (in fact, they'd probably freak at Scenario A too)  There's a big difference between installing an application for a specific purpose such as VNC, vs. random Javascript on websites turning the browser into VNC.  Was there discussion on the con call about what IT department types would think of these two scenarios?

-hadriel

---

From: Randell Jesup <[email-masked]>
Date: Mon, 19 Sep 2011 10:01:24 -0400

> Scenario B, on the other hand, is the other half of VNC and my IT dept. would probably freak if a browser could do this, even with user consent.  (in fact, they'd probably freak at Scenario A too)…

To be honest, no, it wasn't discussed, and it very much should be.  This might get tied
into the more general discussions of "ongoing" permission, and the related topic currently
being worked on in various bodies is permissions for HTML5 "webapps".  This capability
might not even be offered for the user to accept unless installed as part of a webapp
(mirroring the permissions model of installing applications, such as installing VNC).

BTW, your IT department does know that Windows has Remote Desktop and Remote Assistance,
right?  ;-)

-- 
Randell Jesup
[email-masked]

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 19 Sep 2011 07:20:51 -0700

>> browser into VNC. Was there discussion on the con call about what IT
>> department types would think of these two scenarios?
>
> To be honest, no, it wasn't discussed, and it very much should be. This

The issue of "what will the enterprise think" has certainly been raised 
before on this list (albeit in slightly different contexts). See, e.g., 
http://www.ietf.org/mail-archive/web/rtcweb/current/msg00768.html

---

From: Randell Jesup <[email-masked]>
Date: Mon, 19 Sep 2011 10:22:30 -0400

On 9/19/2011 10:20 AM, Timothy B. Terriberry wrote:
> The issue of "what will the enterprise think" has certainly been 
> raised before on this list (albeit in slightly different contexts). 
> See, e.g., 

Yes, though that's a different topic than the security aspect Hadriel 
was asking about (access to screen, mouse, keyboard, etc).  And those 
have not been adequately explored around this use-case that I've proposed.

-- 
Randell Jesup
[email-masked]

---

From: Hadriel Kaplan <[email-masked]>
Date: Tue, 20 Sep 2011 01:23:12 +0000

> BTW, your IT department does know that Windows has Remote Desktop and Remote Assistance,

Yup, and I don't know what they think of that.  My guess is they're ok with RDP because it's a separate well-known TCP listen port number they can block in the firewalls, and afaik doesn't work behind NATs (when the server is behind a NAT) unless you create static port maps.  And RA requires a separate permissions password for the session, I think.

-hadriel

---

From: Jim McEachern <[email-masked]>
Date: Tue, 20 Sep 2011 01:59:41 +0000

I also support including A) in our use cases now, but deferring B) till later.

Jim

---

From: Emil Ivov <[email-masked]>
Date: Tue, 20 Sep 2011 14:11:34 +0200

Hey there,

I support both A and B. I'd also be OK if they were optional (as far as
that's verifiable by remote peers of course).

Cheers,
Emil

-- 
Emil Ivov, Ph.D.                       67000 Strasbourg,
Project Lead                           France
Jitsi
[email-masked]                        PHONE: +33.1.77.62.43.30
http://jitsi.org                       FAX:   +33.1.77.62.47.31

---

From: Emil Ivov <[email-masked]>
Date: Tue, 20 Sep 2011 14:11:42 +0200

Hey Harald,

> this for now (RTCWeb version 1).

I wouldn't be that worried about this. From the feedback we are getting
from our implementation, users don't seem to mind asynchronous even
delivery that much. They simply expect remote control to lag a little
bit when interacting with a remote desktop.

Cheers,
Emil

--
http://jitsi.org

---

From: Emil Ivov <[email-masked]>
Date: Tue, 20 Sep 2011 14:12:21 +0200

На 19.09.11 09:50, Olle E. Johansson написа:
>>> B) Where a remote peer can provide one or more input types such
>>> as mouse and keyboard to control the local system, not only
>>> including the browser, but also other operating system resources.
>>> This clearly can only happen after additional consent, most
>>> likely on a per occasion consent.
>> mixing of events from multiple sources depends strongly on both
>> proper timing/sequencing and reliable delivery). I would like to
>> not address this for now (RTCWeb version 1).
> I think it's a good use case for the data channel. How many such use
> cases do we have? While use case A is quite often handled as a normal
> video stream, use case B is likely something like VNC. This is an
> application that is part of Microsoft Lync as well as the free SIP
> client Blink today.

I don't see both as having separate implementations. We could very well
have browsers stream the desktop as a regular video flow in one
direction and then add to that (or not) user feedback in the opposite
direction. The feedback could go over signalling (which is what we are
doing in Jitsi), over RTP (something like RFC 4733's DTMF) or over
Pseudo TCP.

Either way I think it's something quite important and I don't think
there's a good reason to leave it for later.

I would be happy to work on that.

Cheers,
Emil

-- 
http://jitsi.org

---

From: Stefan Håkansson LK <[email-masked]>
Date: Tue, 20 Sep 2011 14:58:40 +0200

I think we should skip both these use cases for the time being.

I think we have more than enough to design and agree on to get real-time 
audio and video (with recording) and p2p data. There are design choices 
to make and agree on all over the place (from user consent to RTP).

Besides I think that the most obvious (WebEx-like if you want) use case 
for A is document sharing and slide presentations - and we all know that 
there are several such services available already. So the pieces to 
build them are already available.

So my 2 cents say: let's push this into phase 2, and focus on getting 
phase 1 out in a timely manner :) Sorry for not being enthusiastic.

---

From: Lorenzo Miniero <[email-masked]>
Date: Tue, 20 Sep 2011 15:25:02 +0200

+1, let's not put too many things together.

Lorenzo

Il giorno Tue, 20 Sep 2011 14:58:40 +0200
Stefan Håkansson LK <[email-masked]> ha scritto:

> I think we have more than enough to design and agree on to get
> real-time audio and video (with recording) and p2p data. There are
> design choices to make and agree on all over the place (from user
> consent to RTP).
> 
> Besides I think that the most obvious (WebEx-like if you want) use
> case for A is document sharing and slide presentations - and we all
> know that there are several such services available already. So the
> pieces to build them are already available.
> 
> > Screen/Application/Desktop sharing support use case. My take away
> > from the discussion is that this use cases is likely well enough
> > understood to actually start a consensus call now. However, to us
> > WG chairs it was clear that the use case in question actually needs
> > to be split into two parts.
> >
> > A) Where the RTCWEB enabled browser can use a local application
> > window, the whole desktop or a Screen as a media source that can be
> > encoded and transported over the peerConnection for
> > displaying/playback at the peer.
> >
> > B) Where a remote peer can provide one or more input types such as
> > mouse and keyboard to control the local system, not only including
> > the browser, but also other operating system resources. This
> > clearly can only happen after additional consent, most likely on a
> > per occasion consent.
> >
> > conferencing contexts, like in the WEBEX session the Interim
> > meeting was held with where we shared slides. Where the combination
> > of A and B is providing for VNC/Remote desktop.
> >
> > 1) Do you support or object the inclusion of use case A, B or Both
> > in our Use case document?
> >

-- 
Lorenzo Miniero, COB

Meetecho s.r.l.
Web Conferencing and Collaboration Tools
http://www.meetecho.com

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 20 Sep 2011 16:28:18 +0200

I will conclude this call next week on Monday after a week.

Based on the response rate already, I don't see much need for a longer
time interval for the call.

Cheers

Magnus

-- 

Magnus Westerlund

---

From: Emil Ivov <[email-masked]>
Date: Tue, 20 Sep 2011 16:34:15 +0200

> to make and agree on all over the place (from user consent to RTP).

I am not sure I understand the we-already-have-too-much-on-our-plate
argument. If Desktop Sharing is declared optional then the specification
could move at its own pace without blocking other specs or implementations.

> build them are already available.

We already have more than several ways of doing VoIP and video calls,
many of which already work in browsers. Obviously, the argument hasn't
prevented us from starting RTCWEB.

Emil

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 20 Sep 2011 09:08:55 -0600

I think we should do both and we should do it by using SDP to negotiation a session of VNC as defined in RFC 6143. Given we will have a way of setting up a data channel between endpoints, running RFC 6143 on top of it is trivial and does not add much work or complexity. I would propose this as optional to implement.

---

From: Randell Jesup <[email-masked]>
Date: Tue, 20 Sep 2011 11:23:46 -0400

> I would be happy to work on that.
Great!  There is one fundamental question to answer about this use-case:
How does it work in the existing JS security model?

For B) to work, it not only has to accept mouse/keyboard/etc input from
the other side, but it must take that and inject it into local browser or
OS.

The only way I can see to get that to work without a *massive* breach in the
JS security model would be to have it be something JS can't touch directly,
but like a codec is routed (with user permission) to the playback mechanism.
And even there the permission from the user would have to be very carefully
tied to the source.

This would mean that the format for all the data on the wire would need to
be specified and standardized; it can't easily be an application-specific
protocol. (Ok, I can see one painful, indirect way to avoid our building a
remote desktop wire protocol, which is to give a way for the JS app to provide
the message format in a non-executable manner that we could use to transform
the wire data into user-input data.  But that's really no win for us.)  Or
one has to standardize it as a "codec", but that's complicated by not wanting
to deliver it over an unreliable channel.

The alternative would be to have the JS app be (VERY) trusted, which so far
is not the threat model we're using.  Perhaps once again this ties into whatever
security model will be used for user-installed web-apps, since those in many
cases will be taking the place of trusted binary installed apps.

So, looking into this would be great, but I suspect it we can't put it into
the first spec.  It could be a follow-on, perhaps under the webapp security
model.  (My apologies; I know that there's lots of work going on in that
space, but I haven't been following it closely.)

While I'd love to see both A and B in WebRTC 1.0, I just don't see B as being
feasible in this timeframe.  Feel free to show I'm wrong, though!

-- 
Randell Jesup
[email-masked]

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 21 Sep 2011 09:40:07 +0200

> prevented us from starting RTCWEB.

I should have said that "the pieces to build them are already available 
_in browsers without the need for using extensions or plugins_" or 
something similar - I think this is what is put out as the 
differentiating factor in the rtcweb charter.

Anyway, I seem to be rather lonely in my corner of the room, but I 
maintain my opinion that we should defer this to a later stage.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 21 Sep 2011 10:40:45 +0200

For your info / full disclosure / shameless plug:

Today's announcement of "Google+ Hangouts with Extras" includes an 
implementation of window sharing, which indeed encodes a window as a 
video stream and sends it over the network.

Here's the help page:

http://www.google.com/support/plus/bin/static.py?page=guide.cs&guide=1257349&topic=1651691 
<http://www.google.com/support/plus/bin/static.py?page=guide.cs&guide=1257349&topic=1651691>

So we know that we can make one design approach work in our browser 
plugin....

               Harald

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 21 Sep 2011 09:14:42 +0000

Cool.  I like the "Watch YouTube clips together" feature.  It's so romantic.

Actually, the youtube video should be mashed together with this:
http://home.comcast.net/~jps201/mst3k_files/mst3kbackg.jpg
as well as the "Group Chat" feature so folks can do MST3K commentaries of the youtube video they're watching.
Then call the mashup "Bad-AttiTubes".
:)

-hadriel

On Sep 21, 2011, at 4:40 AM, Harald Alvestrand wrote:

> Today's announcement of "Google+ Hangouts with Extras" includes an implementation of window sharing, which indeed encodes a window as a video stream and sends it over the network.
> 
> Here's the help page:
> 
> http://www.google.com/support/plus/bin/static.py?page=guide.cs&guide=1257349&topic=1651691 <http://www.google.com/support/plus/bin/static.py?page=guide.cs&guide=1257349&topic=1651691>
> 
> So we know that we can make one design approach work in our browser plugin....
> 
>              Harald

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 21 Sep 2011 09:03:44 -0700

> maintain my opinion that we should defer this to a later stage.

[BA]  You are not lonely (at least with respect to deferring discussion of scenario B). 

I believe that we can include scenario A within the use case document
and security document, and see how much this affects the requirements and
security issues (I expect not too much). 

However, I am in favor of deferring scenario B rather than adding it.  This
scenario has potentially major security implications, which could slow progress
on other items.  To me, this is a good reason to defer it.  If we keep adding
major work items, the overall schedule will slip, and that is quite dangerous.

---

From: Emil Ivov <[email-masked]>
Date: Wed, 21 Sep 2011 20:49:34 +0200

> major work items, the overall schedule will slip, and that is quite

Why would the schedule slip? If this is an optional extension, as
everyone agrees it should be, how could it delay the main specs and
implementations? Even if there are indeed some unforeseen grave security
issues (which I am not all that sure about), then it they would only
delay release of this extension, wouldn't they?

Emil

--
http://jitsi.org

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 21 Sep 2011 12:01:53 -0700

> delay release of this extension, wouldn't they?

[BA] Even if scenario B is optional, we will still need to do the work to analyze the security implications
and integrate it within the overall architecture.   Unless all work on scenario B were to be separated
out into a separate series of work items with no impact on existing ones (which I believe would require
a charter change), adding scenario B would potentially affect other more basic deliverables.

---

From: Emil Ivov <[email-masked]>
Date: Wed, 21 Sep 2011 21:14:46 +0200

> a charter change), adding scenario B would potentially affect other more

Mmmm ... I am still not sure I am seeing it. I am probably missing
something though, so would you mind being a bit more specific? What
security issues do we expect that would be completely different from
what already needs to be handled with audio and video acquisition?

Emil

-- 
http://jitsi.org

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 30 Sep 2011 11:37:03 +0200

WG,

This is the conclusion of this call for consensus.

I see a consensus for A, but there is a few individuals objecting
against it.

For B the is more significant push back from quite many individuals. The
main arguments being security concerns and the time required to analyze
the security concerns and find a solution for them. The security
concerns include both consent issues and questions of control
(allowing/disallowing) of the functionality within enterprises etc.

Thus my determination is that A shall be included in the use case
document, while B is not included. Editors of the use cases document,
can you please prepare some text for the use case and any derived
requirements that you see.

Best Regards

Magnus Westerlund
WG chair

-- 

Magnus Westerlund

---

From: Stefan Håkansson LK <[email-masked]>
Date: Fri, 30 Sep 2011 11:50:34 +0200

We'll draft something for A.

Stefan for the Editors.
________________________________________
From: Magnus Westerlund [[email-masked]]
Sent: Friday, September 30, 2011 11:37 AM
To: [email-masked]; [email-masked]
Subject: Re: [rtcweb] Call for Consensus on Use Case for Screen/Application/Desktop sharing

WG,

Best Regards

Magnus Westerlund
WG chair

--

Magnus Westerlund

---

From: Stefan Håkansson LK <[email-masked]>
Date: Fri, 30 Sep 2011 12:04:38 +0200

Just a note: we'll not release a new version of the use case doc this week as discussed earlier, instead we'll incorporate this use case in a new version sometime next week.

Stefan
________________________________________
From: Stefan Håkansson LK [[email-masked]]
Sent: Friday, September 30, 2011 11:50 AM
To: Magnus Westerlund; [email-masked]; [email-masked]
Subject: RE: [rtcweb] Call for Consensus on Use Case for Screen/Application/Desktop sharing

Stefan for the Editors.

WG,

Best Regards

Magnus Westerlund
WG chair

--

Magnus Westerlund