Subject: [dispatch] Updated charter/work description for telepresence multi-streams

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Wed, 2 Jun 2010 09:11:59 -0700

Folks,

Here is a version of the work description that addresses the comments
made on the first draft. 

The changes include: 

1.       Charter section, qualification that we are working on SIP-based
systems

2.       Charter section, minor word smithing

3.       Charter section, reference to RAI WG with relevant work

4.       Scope section, clarification of interoperability requirements

5.       Scope section, replacement of second paragraph regarding
treatment of non-video non-audio media types

6.       Goals and Milestones - addition of a Problem Statement draft

Comments welcome

Allyn

Multi-streams for Telepresence Description of Work

Background

One branch of video conferencing has evolved that is focused on
immersive "being there" experience.  Referred to in various ways such as
virtual conferencing, telepresence or media spaces, early systems were
mainly research projects or business systems with limited deployments.
In recent years telepresence systems have seen considerable market
success.  Following the model of early systems, the first wave of
commercial systems have been typically located in specially designed
single-purpose rooms with multiple relatively large displays permitting
life size image reproduction, multiple cameras, encoders, decoders and
microphones.  These systems have several important characteristics that
are different from more traditional video conferencing systems.  

The first difference concerns controlling the visual viewpoint in order
to improve participant nonverbal communication. These systems preserve
essential group meeting characteristics such as eye contact, group
gestures, seating order and spatial audio by carefully orchestrating the
miking and camera angles at each of the sites . This is distinct from
the more traditional approach where the geometric relationship between
media streams is not used to preserve inter-stream communication aspects
such as eye contact and group dynamics.  

A second difference is manipulation of the environment to improve
immersion.  With telepresence systems, cinematographic aspects of the
local environment reproduction are carefully planned including color,
table shape, seating and lighting so that when combined with large high
quality displays, a strong sense of a "trompe l'oeil" or "being there"
immersive experience is created.  Typical video conference systems do
not include these considerations.

As telepresence video systems have become successful in the market,
manufacturers have started exploring delivery of the nonverbal
communication and immersive values of telepresence via smaller, less
expensive and more flexible video conferencing systems for a variety of
venues, such as individual offices, homes and kiosks. These are also
telepresence systems, since the audio and video quality is high enough
to allow clear image reproduction for nonverbal communication, they are
able to send and receive multiple media streams, and large coordinated
multi image displays are available for immersive installations.   As the
industry develops, the line between telepresence and video conferencing
may become blurred as nonverbal communication and immersive
installations become broadly available.

Problem

Although telepresence systems are based on open standards such as RTP,
SIP, H.264, H.323 suite, they cannot easily interoperate with each other
without operator assistance and expensive additional equipment that
translates from one vendor to another. It would be like having to make
sure all parties are on the same equipment (and network) when making a
telephone call.  A major factor in the inability of Telepresence systems
to work with each other is that there is no standard description of the
multiple streams that comprise the media flows. 

For example, in a multiple screen conference, the video and audio
streams sent from remote participants must be understood by receivers so
that they can be presented in a coherent and life-like manner. This
includes the ability to present the remote participants at their true
size for their apparent distance, while maintaining correct eye contact,
gesticular cues, and simultaneously providing a spatial audio sound
stage consistent with the video presentation.  The receiving device that
decides how to display the incoming information needs to understand a
number of variables such as the spatial position of the speaker, the
field of view of the cameras, the camera zoom, which media stream is
related to each of the displays, etc. 

Charter

The Telepresence Multi-Streams work item in DISPATCH is chartered to
define and specify for SIP-based systems the content of media
multi-stream messages and the way these will be transported. 

This work will provide a standard for the exchange of media semantic
information that will foster interoperable end stations and conference
bridges. It will specify  variables that describe the semantics of the
media streams and the recommended behavior to achieve interoperability.

This requires considering current widely deployed use cases, such as
single and multiple screens, multi-point, Scalable Video Coding (SVC),
as well as cases that are expected to be implemented using the protocol
framework produced by this work item.  The methodology for describing
the variables must allow extensibility of the variables, since
telepresence is still a young technology and may have use cases that are
not currently considered.

The work item will identify use cases, define requirements, and define a
method for describing and transporting information about multiple media
streams, including a specification of variables required to support the
use cases. This work item will consider the reuse of existing IETF
protocols and produce an architecture/protocol framework document
describing the protocols required to be implemented to support this
functionality.  The document will identify any enhancements required to
existing protocols as well as describing new protocol(s) for
interoperable multi-streams negotiation that may be required.

Relevant work to be drawn upon has been done in XCON, MMUSIC, AVT, and
FECframe.

Scope

The scope includes both systems that provide a fully immersive
experience, and systems that interwork with them and therefore need to
understand the same multiple stream semantics.

The focus of this work is on audio and video multiple streams.  Other
media types may be considered, however development of methodologies for
them is not within the scope of this work.

Interoperation with standards compliant systems is required, such as
SIP-based video conferencing systems.  However, backwards compatibility
with existing non-standards compliant telepresence systems is not
required.

The group will produce

- Requirements and use cases

- Architectural Framework describing the protocols required to be
implemented to support this functionality and identifying existing
protocol enhancements and new protocol functionality required

- Specification of a new protocol to support telepresence multi-streams
[if required]

Goals and Milestones

Nov 2010 

Use Cases and Requirements to IESG as Informational RFC 

Nov 2010

March 2011 

Problem Statement to IESG as Informational RFC

Architecture to IESG as Informational RFC 

March 2011

Revise Charter [IF new protocol is not required]

Nov 2011 

Submit protocol draft to IESG as Proposed Standard RFC

---

From: Peter Musgrave <[email-masked]>
Date: Wed, 2 Jun 2010 12:39:58 -0400

Hi,

I read this through. I think it describes the work well enough at this
point.

Is it worth narrowing the scope of how media will be transmitted? I assume
it will be RTP but this not explicitly stated.

Regards,

Peter Musgrave

On Wed, Jun 2, 2010 at 12:11 PM, Allyn Romanow (allyn) <[email-masked]>wrote:

>  Folks,
>
> Here is a version of the work description that addresses the comments made
> on the first draft.
>
> The changes include:
>
> 6.       Goals and Milestones – addition of a Problem Statement draft
>
>
>
>
>
> Comments welcome
>
>
>
> Allyn
>
>
>
> *Multi-streams for Telepresence Description of Work*
>
> * *
>
> * *
>
> * *
>
> *Background*
>
> One branch of video conferencing has evolved that is focused on immersive
> “being there” experience.  Referred to in various ways such as virtual
> conferencing, telepresence or media spaces, early systems were mainly
> research projects or business systems with limited deployments.  In recent
> years telepresence systems have seen considerable market success.  Following
> the model of early systems, the first wave of commercial systems have been
> typically located in specially designed single-purpose rooms with multiple
> relatively large displays permitting life size image reproduction, multiple
> cameras, encoders, decoders and microphones.  These systems have several
> important characteristics that are different from more traditional video
> conferencing systems.
>
>
>
> The first difference concerns controlling the visual viewpoint in order to
> improve participant nonverbal communication. These systems preserve
> essential group meeting characteristics such as eye contact, group gestures,
> seating order and spatial audio by carefully orchestrating the miking and
> camera angles at each of the sites . This is distinct from the more
> traditional approach where the geometric relationship between media streams
> is not used to preserve inter-stream communication aspects such as eye
> contact and group dynamics.
>
>
>
> immersion.  With telepresence systems, cinematographic aspects of the local
> environment reproduction are carefully planned including color, table shape,
> seating and lighting so that when combined with large high quality displays,
> a strong sense of a "trompe l'oeil" or "being there" immersive experience is
> created.  Typical video conference systems do not include these
> considerations.
>
>
>
> manufacturers have started exploring delivery of the nonverbal communication
> and immersive values of telepresence via smaller, less expensive and more
> flexible video conferencing systems for a variety of venues, such as
> individual offices, homes and kiosks. These are also  telepresence systems,
> since the audio and video quality is high enough to allow clear image
> reproduction for nonverbal communication, they are able to send and receive
> multiple media streams, and large coordinated multi image displays are
> available for immersive installations.   As the industry develops, the line
> between telepresence and video conferencing may become blurred as nonverbal
> communication and immersive installations become broadly available.
>
>
>
> *Problem*
>
> Although telepresence systems are based on open standards such as RTP, SIP,
> H.264, H.323 suite, they cannot easily interoperate with each other without
> operator assistance and expensive additional equipment that translates from
> one vendor to another. It would be like having to make sure all parties are
> on the same equipment (and network) when making a telephone call.  A major
> factor in the inability of Telepresence systems to work with each other is
> that there is no standard description of the multiple streams that comprise
> the media flows.
>
>
>
> For example, in a multiple screen conference, the video and audio streams
> sent from remote participants must be understood by receivers so that they
> can be presented in a coherent and life-like manner. This includes the
> ability to present the remote participants at their true size for their
> apparent distance, while maintaining correct eye contact, gesticular cues,
> and simultaneously providing a spatial audio sound stage consistent with the
> video presentation.  The receiving device that decides how to display the
> incoming information needs to understand a number of variables such as the
> spatial position of the speaker, the field of view of the cameras, the
> camera zoom, which media stream is related to each of the displays, etc.
>
>
>
> *Charter*
>
> The Telepresence Multi-Streams work item in DISPATCH is chartered to define
> and specify for SIP-based systems the content of media multi-stream messages
> and the way these will be transported.
>
>
>
> This requires considering current widely deployed use cases, such as single
> and multiple screens, multi-point, Scalable Video Coding (SVC), as well as
> cases that are expected to be implemented using the protocol framework
> produced by this work item.  The methodology for describing the variables
> must allow extensibility of the variables, since telepresence is still a
> young technology and may have use cases that are not currently considered.
>
>
>
> streams, including a specification of variables required to support the use
> cases. This work item will consider the reuse of existing IETF protocols and
> produce an architecture/protocol framework document describing the protocols
> required to be implemented to support this functionality.  The document will
> identify any enhancements required to existing protocols as well as
> describing new protocol(s) for interoperable multi-streams negotiation that
> may be required.
>
>
>
> *Scope*
> The scope includes both systems that provide a fully immersive experience,
> and systems that interwork with them and therefore need to understand the
> same multiple stream semantics.
>
>
>
>
>
> The focus of this work is on audio and video multiple streams.  Other media
> types may be considered, however development of methodologies for them is
> not within the scope of this work.
>
>
>
> SIP-based video conferencing systems.  However, backwards compatibility with
> existing non-standards compliant telepresence systems is not required.
>
>
>
> * *
>
>
>
> *The group will produce*
>
> implemented to support this functionality and identifying existing protocol
> enhancements and new protocol functionality required
>
>
>
> - Specification of a new protocol to support telepresence multi-streams [if
> required]
>
>
>
> *Goals and Milestones*
>
> Nov 2010
>
>
>
> _______________________________________________
> dispatch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch
>
>

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Wed, 2 Jun 2010 09:44:52 -0700

Hi Peter,

It's certainly possible imo to explicitly state RTP. We could put it on
the list for changes, if there is no objection.

Allyn

From: Peter Musgrave [mailto:[email-masked]] 
Sent: Wednesday, June 02, 2010 9:40 AM
To: Allyn Romanow (allyn)
Cc: [email-masked]; [email-masked]; Cullen Jennings
(fluffy)
Subject: Re: [dispatch] Updated charter/work description for
telepresence multi-streams

Hi, 

Is it worth narrowing the scope of how media will be transmitted? I
assume it will be RTP but this not explicitly stated.

Regards,

Peter Musgrave

On Wed, Jun 2, 2010 at 12:11 PM, Allyn Romanow (allyn) <[email-masked]>
wrote:

Folks,

The changes include: 

 

Comments welcome

Allyn

 

Background

Problem

Charter

 

Scope

 

 

The group will produce

Goals and Milestones

Nov 2010 

Nov 2010

March 2011 

March 2011

Nov 2011

---

From: Roni Even <[email-masked]>
Date: Tue, 08 Jun 2010 23:43:45 +0300

Hi,

I think this work is important and would like to see it addressed.

The first issue I see is that currently when the SDP offer includes multiple
m-lines of the same media type which typically will have different
semantics, it is not clear how the answerer will select the right streams
and render them correctly.  The content attribute provide some information
about the type of the stream but it does not discuss the application usage
in a way that will help interoperability. I believe that the use case and
requirements will provide more information about the interoperability issues
for this application and I think that this work will allow us to develop
similar solutions for other use cases that use multiple media stream of the
same media type .

Roni Even

From: [email-masked] [mailto:[email-masked]] On Behalf
Of Allyn Romanow (allyn)
Sent: Wednesday, June 02, 2010 7:12 PM
To: [email-masked]; [email-masked]; Cullen Jennings (fluffy)
Subject: [dispatch] Updated charter/work description for telepresence
multi-streams

Folks,

The changes include: 

5.       Scope section, replacement of second paragraph regarding treatment
of non-video non-audio media types

 

Comments welcome

Allyn

 

Background

"being there" experience.  Referred to in various ways such as virtual

A second difference is manipulation of the environment to improve immersion.
With telepresence systems, cinematographic aspects of the local environment
reproduction are carefully planned including color, table shape, seating and
lighting so that when combined with large high quality displays, a strong
sense of a "trompe l'oeil" or "being there" immersive experience is created.
Typical video conference systems do not include these considerations.

Problem

Charter

This work will provide a standard for the exchange of media semantic
information that will foster interoperable end stations and conference
bridges. It will specify  variables that describe the semantics of the media
streams and the recommended behavior to achieve interoperability.  

Scope

 

 

The group will produce

Goals and Milestones

Nov 2010 

Nov 2010

March 2011 

March 2011

Nov 2011