Subject: Re: [payload] Adam Roach's Discuss on draft-ietf-payload-rtp-ancillary-10: (with DISCUSS and COMMENT)

From: Roni Even <[email-masked]>
Date: Sun, 1 Oct 2017 11:25:13 +0000

Hi Thomas,
I think that you should also have in the grouping section a recommendation to use the same persistent CNAME (maybe short term?) for the ANC and video stream see RFC 7022 this is required in RTCP if these RTP streams need to be synchronized (RFC3550)
Roni

> -----Original Message-----
> From: Thomas Edwards [mailto:[email-masked]]
> Sent: יום ד 27 ספטמבר 2017 18:47
> To: Roni Even; Adam Roach; The IESG
> Cc: [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Subject: Re: Adam Roach's Discuss on draft-ietf-payload-rtp-ancillary-10:
> (with DISCUSS and COMMENT)
> 
> Roni, thanks for the feedback on LS!
> 
> I’m still not sure if it is appropriate to include the hierarchical grouping SDP
> example in draft-ietf-payload-rtp-ancillary.  I suspect it could be confusing.
> 
> However, I will note that the Alliance for IP Media Solutions
> (http://aimsalliance.org) Technical Committee is considering work on
> documenting different SDP groupings for live television production scenarios,
> so it is possible that a future I-D will be submitted that could include a range
> of SDP applications in this area, perhaps including documentation of Adam
> Roach’s scenario.
> 
> I can go ahead and upload a draft-ietf-payload-rtp-ancillary-11 based on the
> IESG feedback.
> 
> -Thomas
> 
> --
> Thomas Edwards
> FOX Networks Engineering & Operations
> VP Engineering & Development
> [email-masked]
> +1-310-369-6696
> 10201 W Pico Blvd
> Los Angeles, CA 90035
> 
> 
> On 9/27/17, 4:07 AM, "Roni Even" <[email-masked]> wrote:
> 
>     Hi,
> 
>     I agree that LS is better fit for ANC
> 
> 
> 
>     As for Adam's example it is allowed by RFC5888 to specify
> 
>             a=group:LS V1 V2 M1 M2
> 
>             a=group:LS V1 M1
> 
>             a=group:LS V2  M2
> 
>     this is similar to RFC5956  which replaced RFC4756
> 
> 
> 
>     this is not a clean way but  is allowed otherwise there is a need to specify
> some SDP attribute for specifying dependencies.
> 
> 
> 
>     Roni Even
> 
> 
> 
> 
> 
>     > -----Original Message-----
> 
>     > From: Thomas Edwards [mailto:[email-masked]]
> 
>     > Sent: יום ד 27 ספטמבר 2017 04:13
> 
>     > To: Adam Roach; The IESG
> 
>     > Cc: [email-masked]; [email-masked];
> 
>     > [email-masked]; [email-masked]
> 
>     > Subject: Re: Adam Roach's Discuss on draft-ietf-payload-rtp-ancillary-10:
> 
>     > (with DISCUSS and COMMENT)
> 
>     >
> 
>     > Regarding draft-ietf-payload-rtp-ancillary-10, on further thought I believe
> 
>     > that FID grouping is NOT appropriate for Section 4.1 “Grouping ANC
> Streams
> 
>     > with other Media Streams”.
> 
>     >
> 
>     > RFC 5888 Section 8.4. (“FID Semantics”) states:
> 
>     >
> 
>     >        “It is assumed that the application uses only one codec at a time to
> 
>     > encode the media produced.  This codec MAY change dynamically during
> the
> 
>     > session, but at any particular moment, only one codec is in use.”
> 
>     >
> 
>     > RFC 5888 Section 8.5.1 states:
> 
>     >
> 
>     > 	“FID semantics are useful when the application only uses one codec
> 
>     > at a time.  An application that encodes the same media using different
> 
>     > codecs simultaneously MUST NOT use FID to group those media lines.”
> 
>     >
> 
>     > When we have one video stream and one ancillary data stream, senders
> are
> 
>     > not switching between codecs.  FID really sounds to me like a situation
> where
> 
>     > you have one media stream, but senders could send that stream one of
> 
>     > multiple codecs, but not at the same time.
> 
>     >
> 
>     > Compare with RFC 5888 Section 7:
> 
>     >
> 
>     > 	“Note that LS semantics apply not only to a video stream that has to
> 
>     > be synchronized with an audio stream; the playout of two streams of the
> 
>     > same type can be synchronized as well.”
> 
>     >
> 
>     > Synchronized presentation to the user of an audio, video, and of course
> 
>     > ancillary data flow is generally what we need for production television
> 
>     > workflows.
> 
>     >
> 
>     > draft-ietf-payload-rtp-ancillary-10 Section 4.1 has the example:
> 
>     >
> 
>     >         v=0
> 
>     >         o=Al 123456 11 IN IP4
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__host.example.com&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx
> 6zvFcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=UuOwClXoGCvGP2qksiV3h85H
> NgQRo_Z7b0HgU2qVwzQ&e=
> 
>     >         s=Professional Networked Media Test
> 
>     >         i=A test of synchronized video and ANC data
> 
>     >         t=0 0
> 
>     >         a=group:LS V1 M1
> 
>     >         m=video 50000 RTP/AVP 96
> 
>     >         c=IN IP4 https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__233.252.0.1_255&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zv
> FcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=mDjccdFR8MWnfQcP_xiMhb-
> bh-Wef_2tKhRDyRilYUM&e=
> 
>     >         a=rtpmap:96 raw/90000
> 
>     >         a=fmtp:96 sampling=YCbCr-4:2:2; width=1280; height=720; depth=10
> 
>     >         a=mid:V1
> 
>     >         m=video 50010 RTP/AVP 97
> 
>     >         c=IN IP4 https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__233.252.0.2_255&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zv
> FcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=7e66ewQVAuL1YUq2FzASyEJp
> qw7Tc5p8fO8lZbZ4tUo&e=
> 
>     >         a=rtpmap:97 smpte291/90000
> 
>     >         a=fmtp:97 DID_SDID={0x61,0x02};DID_SDID={0x41,0x05}
> 
>     >         a=mid:M1
> 
>     >
> 
>     > As some have pointed out, RTP timestamps and RTCP provide inherent
> 
>     > synchronization between “m” lines, but of course as RFC 5888 Section 1
> says
> 
>     > without grouping: “When a session description contains more than one
> "m"
> 
>     > line, SDP does not provide any means to express a particular relationship
> 
>     > between two or more of them.”  “LS” grouping makes it pretty clear that
> 
>     > these streams are intended for synchronized presentation.
> 
>     >
> 
>     > An alternative to “LS” grouping is to drop all grouping in the example, and
> be
> 
>     > unclear about what these “m” lines are doing together in the same file:
> 
>     >
> 
>     >         v=0
> 
>     >         o=Al 123456 11 IN IP4
> https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__host.example.com&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx
> 6zvFcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=UuOwClXoGCvGP2qksiV3h85H
> NgQRo_Z7b0HgU2qVwzQ&e=
> 
>     >         s=Professional Networked Media Test
> 
>     >         i=A test of synchronized video and ANC data
> 
>     >         t=0 0
> 
>     >         m=video 50000 RTP/AVP 96
> 
>     >         c=IN IP4 https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__233.252.0.1_255&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zv
> FcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=mDjccdFR8MWnfQcP_xiMhb-
> bh-Wef_2tKhRDyRilYUM&e=
> 
>     >         a=rtpmap:96 raw/90000
> 
>     >         a=fmtp:96 sampling=YCbCr-4:2:2; width=1280; height=720; depth=10
> 
>     >         m=video 50010 RTP/AVP 97
> 
>     >         c=IN IP4 https://urldefense.proofpoint.com/v2/url?u=http-
> 3A__233.252.0.2_255&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zv
> FcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=7e66ewQVAuL1YUq2FzASyEJp
> qw7Tc5p8fO8lZbZ4tUo&e=
> 
>     >         a=rtpmap:97 smpte291/90000
> 
>     >         a=fmtp:97 DID_SDID={0x61,0x02};DID_SDID={0x41,0x05}
> 
>     >
> 
>     > But I’m still more comfortable with keeping the “LS” grouping in the
> example
> 
>     > unless there are objections.
> 
>     >
> 
>     > This brings us to the main point of Adam Roach’s DISCUSS item, how do
> you
> 
>     > provide multi-level hierarchical grouping in SDP.  Imagine a group of (a
> group
> 
>     > of video and ancillary data) and (a group of video and ancillary data).
> There
> 
>     > actually is a case of this in production television, the multiviewer (e.g. see
> 
>     > https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.blackmagicdesign.com_products_multiview&d=DwIGaQ&c=uw6T
> Lu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-
> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=mmtyLXtya
> WHEg3JYcq3xSVLXzRqkIVgWAJDgqyIgreU&s=IdKozSZWbohK0Fg_CvZcmeNO
> 6JxjGZvwknrCRLaQVX8&e=).
> 
>     >
> 
>     > Unfortunately, I really can’t find any SDP mechanism to describe multi-
> level
> 
>     > hierarchy.  RFC 4796 “The Session Description Protocol (SDP) Content
> 
>     > Attribute” is about the best I can find, where synchronized multimedia
> (such
> 
>     > as “slides”, “speaker”, “main”, etc.) is presented.  But this is a very limited
> set
> 
>     > of values.  RFC 5583 “Signaling Media Decoding Dependency in the
> Session
> 
>     > Description Protocol (SDP)” has various dependencies of streams, but
> this is
> 
>     > for layered or multiple descriptive media coding.  Neither of these seems
> 
>     > appropriate.
> 
>     >
> 
>     > Perhaps this issue needs to be brought up in Multiparty Multimedia
> Session
> 
>     > Control (mmusic) WG, but I’m not sure that it is wise to hold up the ANC
> RTP
> 
>     > Payload over this.
> 
>     >
> 
>     > Thanks everyone!
> 
>     >
> 
>     > -Thomas
> 
>     >
> 
>     > --
> 
>     > Thomas Edwards
> 
>     > FOX Networks Engineering & Operations
> 
>     > VP Engineering & Development
> 
>     > [email-masked]
> 
>     > +1-310-369-6696
> 
>     > 10201 W Pico Blvd
> 
>     > Los Angeles, CA 90035
> 
>     >
> 
>     >
> 
>     >
> 
>     >
> 
> 
> 
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 17:55:53 +0100

> On 29 Sep 2017, at 22:00, Adam Roach <[email-masked]> wrote:
> On 9/26/17 8:13 PM, Thomas Edwards wrote:
>> Regarding draft-ietf-payload-rtp-ancillary-10, on further thought I believe that FID grouping is NOT appropriate for Section 4.1 “Grouping ANC Streams with other Media Streams”.
>> 
>>        “It is assumed that the application uses only one codec at a time to encode the media produced.  This codec MAY change dynamically during the session, but at any particular moment, only one codec is in use.”
>> 
>> 	“FID semantics are useful when the application only uses one codec at a time.  An application that encodes the same media using different codecs simultaneously MUST NOT use FID to group those media lines.”
>> 
>> When we have one video stream and one ancillary data stream, senders are not switching between codecs.  FID really sounds to me like a situation where you have one media stream, but senders could send that stream one of multiple codecs, but not at the same time.
> 
> In this case, where you have data and meta-data, it's arguably two different codecs; and, since they don't represent the same data, you're encoding with one codec or the other, based on whether you're sending video data or meta-data. From that perspective, I don't see the use of FID as being outside the spirit of what is intended.
> 
> That said, if the WG agrees with your evaluation that FID isn't quite the right fit, I'm happy for y'all to come up with some alternate solution...

I tend to agree with Adam that FID is appropriate to group the ancillary data with its associated media stream. The metadata in the ancillary stream is conceptually part of the media stream, which matches the definition of FID. They MUST be sent with the same CNAME, to ensure they can be synchronised, but that doesn’t require any explicit signalling.

You can use LS signalling in addition to FID if two streams, with their associated ancillary streams, need to be synchronised. 

-- 
Colin Perkins
https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 17:59:20 +0100

Roni,

The requirement for RTP, in general, is that streams that are to be synchronised MUST have the same RTCP CNAME. It might be appropriate to remind readers of that, but it’s independent on the grouping framework. 

How that CNAME is chosen doesn’t matter for the purposes of synchronisation, provided the SSRCs that are to be synchronised use the same CNAME. The suggestions in RFC 7022 are appropriate, but there’s no reason to mandate them, or require a particular one of those suggestions be used.

Colin

> On 1 Oct 2017, at 12:25, Roni Even <[email-masked]> wrote:
> 
> Hi Thomas,
> _______________________________________________
> payload mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/payload

-- 
Colin Perkins
https://csperkins.org/

---

From: Roni Even <[email-masked]>
Date: Tue, 3 Oct 2017 17:53:40 +0000

Hi Colin,
I think that the question is if the video and anc are decoded at the same time. In RFC 5888 the case when DTMF is both in the audio and as DTMF tones (section 8.5.1 ) FID cannot be used only if audio is muted when DTMF tones are sent.
So the question is if video is muted when ANC is sent (only one decode at a specific time). I think that in term of time they are both part of the same video frame so they have the same time so I am not sure if FID can be used. Maybe Thomas can provide is view here if they are decodec in parallel or one after the other?
And yes they MUST have the same CNAME regardless of the grouping specified
Roni

> From: Colin Perkins [mailto:[email-masked]]
> Sent: יום ג 03 אוקטובר 2017 19:56
> To: Adam Roach
> Cc: Thomas Edwards; The IESG; [email-masked]; draft-ietf-payload-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [payload] Adam Roach's Discuss on draft-ietf-payload-rtp-
> ancillary-10: (with DISCUSS and COMMENT)
> 
> that FID grouping is NOT appropriate for Section 4.1 “Grouping ANC Streams
> encode the media produced.  This codec MAY change dynamically during the
> are not switching between codecs.  FID really sounds to me like a situation
> where you have one media stream, but senders could send that stream one
> of multiple codecs, but not at the same time.
> >
> > In this case, where you have data and meta-data, it's arguably two
> different codecs; and, since they don't represent the same data, you're
> encoding with one codec or the other, based on whether you're sending
> video data or meta-data. From that perspective, I don't see the use of FID as
> being outside the spirit of what is intended.
> >
> > That said, if the WG agrees with your evaluation that FID isn't quite the
> right fit, I'm happy for y'all to come up with some alternate solution...
> 
> I tend to agree with Adam that FID is appropriate to group the ancillary data
> with its associated media stream. The metadata in the ancillary stream is
> conceptually part of the media stream, which matches the definition of FID.
> They MUST be sent with the same CNAME, to ensure they can be
> synchronised, but that doesn’t require any explicit signalling.
> 
> You can use LS signalling in addition to FID if two streams, with their
> associated ancillary streams, need to be synchronised.
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> 
>

---

From: Roni Even <[email-masked]>
Date: Tue, 3 Oct 2017 18:00:52 +0000

Hi Colin,
So maybe the right direction is that there is no need for grouping at all in this case and  just use the same CNAME in RTCP.
Roni

> Sent: יום ג 03 אוקטובר 2017 19:59
> To: Roni Even
> Cc: Thomas Edwards; Adam Roach; [email-masked]; draft-ietf-
> [email-masked]; [email-masked]; [email-masked]
> The requirement for RTP, in general, is that streams that are to be
> synchronised MUST have the same RTCP CNAME. It might be appropriate to
> remind readers of that, but it’s independent on the grouping framework.
> 
> How that CNAME is chosen doesn’t matter for the purposes of
> synchronisation, provided the SSRCs that are to be synchronised use the
> same CNAME. The suggestions in RFC 7022 are appropriate, but there’s no
> reason to mandate them, or require a particular one of those suggestions be
> used.
> 
> Colin
> 
> 
> 
> > I think that you should also have in the grouping section a
> > recommendation to use the same persistent CNAME (maybe short term?)
> > for the ANC and video stream see RFC 7022 this is required in RTCP if
> > these RTP streams need to be synchronized (RFC3550) Roni
> >
> >
> >> Cc: [email-masked];
> >> [email-masked];
> >> I’m still not sure if it is appropriate to include the hierarchical
> >> grouping SDP example in draft-ietf-payload-rtp-ancillary.  I suspect it could
> be confusing.
> >>
> >> documenting different SDP groupings for live television production
> >> scenarios, so it is possible that a future I-D will be submitted that
> >> could include a range of SDP applications in this area, perhaps
> >> including documentation of Adam Roach’s scenario.
> >>
> >> I can go ahead and upload a draft-ietf-payload-rtp-ancillary-11 based
> >> on the IESG feedback.
> >>
> >> -Thomas
> >>
> >> --
> >> Thomas Edwards
> >> FOX Networks Engineering & Operations VP Engineering & Development
> >> [email-masked]
> >> +1-310-369-6696
> >> 10201 W Pico Blvd
> >> Los Angeles, CA 90035
> >>
> >>
> >>    this is not a clean way but  is allowed otherwise there is a need
> >> to specify some SDP attribute for specifying dependencies.
> >>
> >>
> >>
> >>    Roni Even
> >>
> >>
> >>
> >>
> >>
> >>> Cc: [email-masked];
> >>> [email-masked];
> >>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 19:46:22 +0100

No, I think it’s orthogonal to grouping. If the CNAME was sufficient, then we wouldn’t need the LS grouping method.

Colin

> On 3 Oct 2017, at 19:00, Roni Even <[email-masked]> wrote:
> 
> Hi Colin,

-- 
Colin Perkins
https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 19:51:05 +0100

Hi Roni,

I tend to think the “decoded at the sane time” issue is over-interpreting the wording of RFC 5888. The metadata and the media to “form a media flow”, which is the definition of FID.

Colin

> On 3 Oct 2017, at 18:53, Roni Even <[email-masked]> wrote:
> 
> Hi Colin,

-- 
Colin Perkins
https://csperkins.org/

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 3 Oct 2017 21:55:40 +0300

I will repeat myself and say I think the FID grouping is the right
thing to do here unless one is willing to specify a new grouping
semantics for ANC data.

---

From: Roni Even <[email-masked]>
Date: Tue, 3 Oct 2017 20:40:41 +0000

Hi Colin,
I re-read RFC5888 and tried to understand what is the difference between FID and LS
I was trying to understand what is a flow and from section 8.3 

"The previous examples show that the definition of a media stream in
   [RFC2326] does not cover some scenarios.  It cannot be assumed that a
   single media instance maps into a single RTP session.  Therefore, we
   introduce the definition of a media flow:

      A media flow consists of a single media instance, e.g., an audio
      stream or a video stream as well as a single whiteboard or shared
      application group.  When using RTP, a media flow comprises one or
      more RTP sessions."

It looks to me that a media flow is a SINGLE media stream that spans over multiple RTP sessions (the term media instance is a what we call a media stream in the taxonomy). Section 8.5 (8.5.1) enhance this by saying when FID cannot be used " FID semantics are useful when the application only uses one codec at   a time"  which again describe a flow as a single media stream.

I am not sure how ANC and the video relates to each other this is what I am asking Thomas to be more clear. If the video and ANC are one stream over multiple RTP sessions FID is correct otherwise I think the LS should be used.
My reading on  the ancillary draft is that ANC and the video are what we call a single RTP stream but this is not what Thomas says.

Roni

> Sent: יום ג 03 אוקטובר 2017 21:51
> To: Roni Even
> Cc: Adam Roach; Thomas Edwards; The IESG; [email-masked]; draft-
> [email-masked]; [email-masked]; [email-masked]
> I tend to think the “decoded at the sane time” issue is over-interpreting the
> wording of RFC 5888. The metadata and the media to “form a media flow”,
> which is the definition of FID.
> 
> Colin
> 
> 
> > I think that the question is if the video and anc are decoded at the same
> time. In RFC 5888 the case when DTMF is both in the audio and as DTMF
> tones (section 8.5.1 ) FID cannot be used only if audio is muted when DTMF
> tones are sent.
> > So the question is if video is muted when ANC is sent (only one decode at a
> specific time). I think that in term of time they are both part of the same
> video frame so they have the same time so I am not sure if FID can be used.
> Maybe Thomas can provide is view here if they are decodec in parallel or one
> after the other?
> > And yes they MUST have the same CNAME regardless of the grouping
> > specified Roni
> >
> >
> >> Cc: Thomas Edwards; The IESG; [email-masked];
> >> draft-ietf-payload- [email-masked]; [email-masked];
> >> [email-masked]
> >> Streams with other Media Streams”.
> >>>>
> >> the session, but at any particular moment, only one codec is in use.”
> >>>>
> >> at a time.  An application that encodes the same media using
> >> different codecs simultaneously MUST NOT use FID to group those media
> lines.”
> >>>>
> >> are not switching between codecs.  FID really sounds to me like a
> >> situation where you have one media stream, but senders could send
> >> that stream one of multiple codecs, but not at the same time.
> >>>
> >> different codecs; and, since they don't represent the same data,
> >> you're encoding with one codec or the other, based on whether you're
> >> sending video data or meta-data. From that perspective, I don't see
> >> the use of FID as being outside the spirit of what is intended.
> >>>
> >> I tend to agree with Adam that FID is appropriate to group the
> >> ancillary data with its associated media stream. The metadata in the
> >> ancillary stream is conceptually part of the media stream, which matches
> the definition of FID.
> >> You can use LS signalling in addition to FID if two streams, with
> >> their associated ancillary streams, need to be synchronised.
> >>
> >> --
> >> Colin Perkins
> >> https://csperkins.org/
> >>
> >>
> >>
> >
> 
> 
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> 
>

---

From: Roni Even <[email-masked]>
Date: Tue, 3 Oct 2017 21:01:37 +0000

Colin,
I re-read RFC5888 section 7 and 7.1. I think that LS for two RTP streams that need to be synchronized will work without LS. The LS is needed for case when grouping m-lines that  are not RTP, for example an RTP with a m=application or like in section 7.1 you have three streams probably from the same source but only two must be synchronized. For the case where there are  for example one video and one audio streams that will be synchronized based on the CNAME, I am not sure what is the value of LS except that it provides signaling plane information that these are the audio and video that must be synchronized even if they do not have the same CNAME and how to synchronize is an application issue (for example the receiver can delay the audio ,  use case  audio mixing in the video switching MCU topology )
Roni

> Sent: יום ג 03 אוקטובר 2017 21:46
> To: Roni Even
> > So maybe the right direction is that there is no need for grouping at all in
> this case and  just use the same CNAME in RTCP.
> > Roni
> >
> >> synchronised MUST have the same RTCP CNAME. It might be appropriate
> >> to remind readers of that, but it’s independent on the grouping
> framework.
> >>
> >> synchronisation, provided the SSRCs that are to be synchronised use
> >> the same CNAME. The suggestions in RFC 7022 are appropriate, but
> >> there’s no reason to mandate them, or require a particular one of
> >> those suggestions be used.
> >>
> >> Colin
> >>
> >>
> >>
> >>> for the ANC and video stream see RFC 7022 this is required in RTCP
> >>> if these RTP streams need to be synchronized (RFC3550) Roni
> >>>
> >>>
> >>>> (http://aimsalliance.org) Technical Committee is considering work
> >>>> on documenting different SDP groupings for live television
> >>>> production scenarios, so it is possible that a future I-D will be
> >>>> submitted that could include a range of SDP applications in this
> >>>> area, perhaps including documentation of Adam Roach’s scenario.
> >>>>
> >>>> I can go ahead and upload a draft-ietf-payload-rtp-ancillary-11
> >>>> based on the IESG feedback.
> >>>>
> >>>> -Thomas
> >>>>
> >>>> --
> >>>> Thomas Edwards
> >>>>> different
> >>>>> synchronization between “m” lines, but of course as RFC 5888
> >>>>> Section
> >>>>> 1
> >>>> says
> >>>>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 22:41:40 +0100

Roni,

LS is unnecessary, yes. Some people like signalling.

Colin

> On 3 Oct 2017, at 22:01, Roni Even <[email-masked]> wrote:
> 
> Colin,

-- 
Colin Perkins
https://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 22:45:36 +0100

> On 3 Oct 2017, at 21:40, Roni Even <[email-masked]> wrote:
> 
> Hi Colin,

My understanding is that the video and ancillary data are the same stream. In broadcast TV the ancillary data is sent in-band, in the blanking intervals.

> My reading on  the ancillary draft is that ANC and the video are what we call a single RTP stream but this is not what Thomas says.

I expect one could send the ancillary data using the same SSRC, and the same 5-tuple, as the media data (in the same way comfort noise and speech data can be sent), in which case you don’t need the grouping. If they’re sent as separate RTP streams, with separate m= lines, then you need grouping. I believe FID makes sense for this.

-- 
Colin Perkins
https://csperkins.org/

---

From: Roni Even <[email-masked]>
Date: Tue, 3 Oct 2017 22:19:05 +0000

Colin,
We still need some feedback from Thomas, in a previous email response to Adam he gave a reason for using multiple RTP sessions:

"The challenge is that the use case of the Ancillary I-D within the SMPTE ST 2110 architecture would generally require separate network destination addresses for the different elements (video, audio, ancillary data) so they can be flexibly composed within broadcast plants at the network level.  Your alternative SDP example appears to place both the video and the ancillary data onto the same destination address (233.252.0.1).  Is there an appropriate SDP mechanism to communicate the intimate relationship between the RTP streams, but also have them transmitted on separate destination IP addresses (and synchronized)?

(In production practice, this may not be a significant issue as the RTP streams to be combined by IP receivers for conversion to SDI/HDMI or for input to a distribution encoder may be specified by a control system directly to the receiver using an API that isn’t just an SDP file.)"

I want to assume that by different destination addresses it refers to multicast addresses like above and not to different unicast IP addresses. 
You are right that for unicast it will make sense to use the same RTP session with the same SSRC.

Roni

> Sent: יום ד 04 אוקטובר 2017 00:46
> To: Roni Even
> > I re-read RFC5888 and tried to understand what is the difference
> > between FID and LS I was trying to understand what is a flow and from
> > section 8.3
> >
> > It looks to me that a media flow is a SINGLE media stream that spans over
> multiple RTP sessions (the term media instance is a what we call a media
> stream in the taxonomy). Section 8.5 (8.5.1) enhance this by saying when FID
> cannot be used " FID semantics are useful when the application only uses
> one codec at   a time"  which again describe a flow as a single media stream.
> >
> > I am not sure how ANC and the video relates to each other this is what I am
> asking Thomas to be more clear. If the video and ANC are one stream over
> multiple RTP sessions FID is correct otherwise I think the LS should be used.
> 
> My understanding is that the video and ancillary data are the same stream. In
> broadcast TV the ancillary data is sent in-band, in the blanking intervals.
> 
> > My reading on  the ancillary draft is that ANC and the video are what we call
> a single RTP stream but this is not what Thomas says.
> 
> 
> I expect one could send the ancillary data using the same SSRC, and the same
> 5-tuple, as the media data (in the same way comfort noise and speech data
> can be sent), in which case you don’t need the grouping. If they’re sent as
> separate RTP streams, with separate m= lines, then you need grouping. I
> believe FID makes sense for this.
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> 
>

---

From: Colin Perkins <[email-masked]>
Date: Tue, 3 Oct 2017 23:50:13 +0100

Roni,

I said nothing about multicast vs unicast, since I don’t think that distinction is important. 

Sending the streams on separate m= lines or sending using a single a m= line both work. The draft doesn’t need to discuss this, since it’s standard RTP as would apply to any RTP session. If you use separate m= lines, then you need a grouping mechanism. The choice of grouping mechanism is what we’re discussing.

Colin

> On 3 Oct 2017, at 23:19, Roni Even <[email-masked]> wrote:
> 
> Colin,

-- 
Colin Perkins
https://csperkins.org/

---

From: Roni Even <[email-masked]>
Date: Wed, 4 Oct 2017 16:26:58 +0000

Hi Colin,
I agree.
There are two cases. The assumption is that both video and ANC have the same source and have same CNAME (Thomas - is this correct?)

1. Both video and ANC are in the same RTP session (same m-line)  will be synchronized. 

The second is the one we are discussing:
2. The video and ANC are in separate RTP sessions (two m-lines). If both are a single flow (single RTP stream) should group using FID. If they are not a single RTP stream (flow) should use LS.

I can agree that it looks like a single flow (FID) but want clarification from the authors (Thomas)

Roni

> Sent: יום ד 04 אוקטובר 2017 01:50
> To: Roni Even
> Sending the streams on separate m= lines or sending using a single a m= line
> both work. The draft doesn’t need to discuss this, since it’s standard RTP as
> would apply to any RTP session. If you use separate m= lines, then you need
> a grouping mechanism. The choice of grouping mechanism is what we’re
> discussing.
> 
> Colin
> 
> 
> 
> 
> > We still need some feedback from Thomas, in a previous email response to
> Adam he gave a reason for using multiple RTP sessions:
> >
> > "The challenge is that the use case of the Ancillary I-D within the SMPTE ST
> 2110 architecture would generally require separate network destination
> addresses for the different elements (video, audio, ancillary data) so they
> can be flexibly composed within broadcast plants at the network level.  Your
> alternative SDP example appears to place both the video and the ancillary
> data onto the same destination address (233.252.0.1).  Is there an
> appropriate SDP mechanism to communicate the intimate relationship
> between the RTP streams, but also have them transmitted on separate
> destination IP addresses (and synchronized)?
> >
> > (In production practice, this may not be a significant issue as the RTP
> streams to be combined by IP receivers for conversion to SDI/HDMI or for
> input to a distribution encoder may be specified by a control system directly
> to the receiver using an API that isn’t just an SDP file.)"
> >
> > I want to assume that by different destination addresses it refers to
> multicast addresses like above and not to different unicast IP addresses.
> >> Cc: Adam Roach; Thomas Edwards; The IESG; [email-masked];
> >> draft- [email-masked]; [email-masked];
> >> [email-masked]
> >>> between FID and LS I was trying to understand what is a flow and
> >>> from section 8.3
> >>>
> >>> [RFC2326] does not cover some scenarios.  It cannot be assumed that
> >>> a  single media instance maps into a single RTP session.  Therefore,
> >>> we  introduce the definition of a media flow:
> >>>
> >> multiple RTP sessions (the term media instance is a what we call a
> >> media stream in the taxonomy). Section 8.5 (8.5.1) enhance this by
> >> saying when FID cannot be used " FID semantics are useful when the
> application only uses
> >> asking Thomas to be more clear. If the video and ANC are one stream
> >> over multiple RTP sessions FID is correct otherwise I think the LS should be
> used.
> >>
> >> My understanding is that the video and ancillary data are the same
> >> stream. In broadcast TV the ancillary data is sent in-band, in the blanking
> intervals.
> >>
> >> I expect one could send the ancillary data using the same SSRC, and
> >> the same 5-tuple, as the media data (in the same way comfort noise
> >> and speech data can be sent), in which case you don’t need the
> >> grouping. If they’re sent as separate RTP streams, with separate m=
> >> lines, then you need grouping. I believe FID makes sense for this.
> >>
> >> --
> >> Colin Perkins
> >> https://csperkins.org/
> >>
> >>
> >>
> >
> 
> 
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> 
>

---

From: Thomas Edwards <[email-masked]>
Date: Tue, 10 Oct 2017 00:06:39 +0000

I will note that I have uploaded a new version, draft-ietf-payload-rtp-ancillary-11, to address most of the comments from the IESG evaluation.

draft-ietf-payload-rtp-ancillary-11, Section 4.1 “Grouping ANC Streams with other Media Streams” has two examples.

In the first SDP example, two “m=” lines are present (with each having a different “c=” line), one for uncompressed video, and one for ancillary data.  The two RTP sessions are identified with RFC 5888 Media Stream Identification Attributes (mid), and are grouped together using Lip Synchronization (LS), in a fashion similar to the Lip Synchronization of video and audio streams in RFC 5888 Section 7.1 “Example of LS”.

In the second SDP example, a single “m=” line is present without any grouping, implying any synchronization would be from SSRC/CNAME linkage and/or RTCP.

The most likely implementation of this RTP payload in professional video production would be where multiple endpoints are contributing elementary streams of media: video, audio channels or channel groups, and SMPTE ST 291-1 ancillary data sources.  Those endpoints (microphones, cameras, Closed Caption generators, etc.) are all likely to be different, thus with different SSRCs and CNAMEs, and transmitting to separate multicast IP addresses and UDP ports.  However, they will likely be transmitted with a commonly synchronized clock as per RFC 7273 “RTP Clock Source Signalling”, most likely IEEE 1588 PTP.  

The individual elementary media streams will be composed together by receivers as required for production.  For example, for an English language service, an English audio channel group of AES67/RFC 3190 RTP audio would be composed together with the RFC 4175 video and the ST 291-1 ancillary data flow of English language Closed Captions ancillary data and SMPTE 2016-1 AFD ancillary data.    Meanwhile the Spanish language service would have the same video and AFD ancillary data, but would have Spanish language audio and Spanish Closed Captions.  Likely it would be a distribution quality encoder (perhaps H.264 encoder) that subscribes to the different streams in order to create the final service.

Every time I read about “FID” in RFC 5888, I am struck by the sentence in Section 8 “Flow Identification (FID)” that says “there are situations where a single media instance (e.g., an audio stream or a video stream) is sent using more than one RTP session,” which I believe is not the case here.  For example, video is one media instance, and ancillary data is a second media instance.  You can’t “add” video and ancillary data together, like you can add together voice audio and DTMF tone audio (as in RFC 5888 Section 8.2).  

RFC 5888 Section 8.4 “FID Semantics” also states: “It is assumed that the application uses only one codec at a time to encode the media produced.  This codec MAY change dynamically during the session, but at any particular moment, only one codec is in use.”  Again, this does not seem relevant to the SDP description of multiple RTP streams of video, audio, and ancillary data intended for synchronized presentation.  All three “codecs” are in use at all times for their particular media stream.  And in particular regarding draft-ietf-payload-rtp-ancillary, there is only one “codec” ever used, the encapsulation of SMPTE ST 291-1 ancillary data packets into RTP.

I could really use the help of the IESG to see how we can move forward regarding the open DISCUSS item on draft-ietf-payload-rtp-ancillary.

(I will note that SMPTE is eagerly awaiting the publishing of this RFC for use as a normative reference in the upcoming ST 2110-40 standard, see: https://www.smpte.org/st-2110).

Thanks!

-Thomas Edwards
 FOX

On 10/4/17, 9:27 AM, "Roni Even" <[email-masked]> wrote:

    Hi Colin,
    
    I agree.

    

    

    

    
    Roni

    

    

    

    
    
    > I said nothing about multicast vs unicast, since I don’t think that distinction is
    
    > important.

    

    

    

    

    

    

    

    
    > data onto the same destination address (https://urldefense.proofpoint.com/v2/url?u=http-3A__233.252.0.1&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=srzDHDG2LTcTBJexIDta4-p9142SYkQ2gDH3S8PMdZg&e=).  Is there an

    

    

    

    
    > > You are right that for unicast it will make sense to use the same RTP session
    
    > with the same SSRC.

    

    

    

    > >> Subject: Re: [payload] Adam Roach's Discuss on
    
    > >> draft-ietf-payload-rtp-

    

    

    

    

    

    

    
    > >>> It looks to me that a media flow is a SINGLE media stream that spans
    
    > >>> over

    

    > >> one codec at   a time"  which again describe a flow as a single media
    
    > stream.

    > >>> I am not sure how ANC and the video relates to each other this is
    
    > >>> what I am

    

    

    
    > >>> My reading on  the ancillary draft is that ANC and the video are
    
    > >>> what we call

    

    

    

    
    > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__csperkins.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRfcPlYZEmqGCYYi9rf9Wc7dYRzxspsY&e=

    

    

    
    
    > https://urldefense.proofpoint.com/v2/url?u=https-3A__csperkins.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRfcPlYZEmqGCYYi9rf9Wc7dYRzxspsY&e=

---

From: "Ali C. Begen" <[email-masked]>
Date: Tue, 10 Oct 2017 09:44:40 +0300

Hi Thomas

Please go ahead and submit a revision. Then reach out to the individual
DISCUSS holders to clear theirs. And then lets focus on Adam's discuss.

Thanks
-acbegen (your payload wg co-chair)

On Tue, Oct 10, 2017 at 3:06 AM, Thomas Edwards <[email-masked]>
wrote:

> I will note that I have uploaded a new version, draft-ietf-payload-rtp-ancillary-11,
> to address most of the comments from the IESG evaluation.
>
> draft-ietf-payload-rtp-ancillary-11, Section 4.1 “Grouping ANC Streams
> with other Media Streams” has two examples.
>
> In the first SDP example, two “m=” lines are present (with each having a
> different “c=” line), one for uncompressed video, and one for ancillary
> data.  The two RTP sessions are identified with RFC 5888 Media Stream
> Identification Attributes (mid), and are grouped together using Lip
> Synchronization (LS), in a fashion similar to the Lip Synchronization of
> video and audio streams in RFC 5888 Section 7.1 “Example of LS”.
>
> In the second SDP example, a single “m=” line is present without any
> grouping, implying any synchronization would be from SSRC/CNAME linkage
> and/or RTCP.
>
> The most likely implementation of this RTP payload in professional video
> production would be where multiple endpoints are contributing elementary
> streams of media: video, audio channels or channel groups, and SMPTE ST
> 291-1 ancillary data sources.  Those endpoints (microphones, cameras,
> Closed Caption generators, etc.) are all likely to be different, thus with
> different SSRCs and CNAMEs, and transmitting to separate multicast IP
> addresses and UDP ports.  However, they will likely be transmitted with a
> commonly synchronized clock as per RFC 7273 “RTP Clock Source Signalling”,
> most likely IEEE 1588 PTP.
>
> The individual elementary media streams will be composed together by
> receivers as required for production.  For example, for an English language
> service, an English audio channel group of AES67/RFC 3190 RTP audio would
> be composed together with the RFC 4175 video and the ST 291-1 ancillary
> data flow of English language Closed Captions ancillary data and SMPTE
> 2016-1 AFD ancillary data.    Meanwhile the Spanish language service would
> have the same video and AFD ancillary data, but would have Spanish language
> audio and Spanish Closed Captions.  Likely it would be a distribution
> quality encoder (perhaps H.264 encoder) that subscribes to the different
> streams in order to create the final service.
>
> Every time I read about “FID” in RFC 5888, I am struck by the sentence in
> Section 8 “Flow Identification (FID)” that says “there are situations where
> a single media instance (e.g., an audio stream or a video stream) is sent
> using more than one RTP session,” which I believe is not the case here.
> For example, video is one media instance, and ancillary data is a second
> media instance.  You can’t “add” video and ancillary data together, like
> you can add together voice audio and DTMF tone audio (as in RFC 5888
> Section 8.2).
>
> RFC 5888 Section 8.4 “FID Semantics” also states: “It is assumed that the
> application uses only one codec at a time to encode the media produced.
> This codec MAY change dynamically during the session, but at any particular
> moment, only one codec is in use.”  Again, this does not seem relevant to
> the SDP description of multiple RTP streams of video, audio, and ancillary
> data intended for synchronized presentation.  All three “codecs” are in use
> at all times for their particular media stream.  And in particular
> regarding draft-ietf-payload-rtp-ancillary, there is only one “codec”
> ever used, the encapsulation of SMPTE ST 291-1 ancillary data packets into
> RTP.
>
> I could really use the help of the IESG to see how we can move forward
> regarding the open DISCUSS item on draft-ietf-payload-rtp-ancillary.
>
> (I will note that SMPTE is eagerly awaiting the publishing of this RFC for
> use as a normative reference in the upcoming ST 2110-40 standard, see:
> https://www.smpte.org/st-2110).
>
> Thanks!
>
> -Thomas Edwards
>  FOX
>
>     There are two cases. The assumption is that both video and ANC have
> the same source and have same CNAME (Thomas - is this correct?)
>
>
>
>     2. The video and ANC are in separate RTP sessions (two m-lines). If
> both are a single flow (single RTP stream) should group using FID. If they
> are not a single RTP stream (flow) should use LS.
>
>
>
> draft-
>     > data onto the same destination address (https://urldefense.
> proofpoint.com/v2/url?u=http-3A__233.252.0.1&d=DwIGaQ&c=
> uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=
> lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_
> tU3divd4ks6mmXPhk5WmodLULHUPY&s=srzDHDG2LTcTBJexIDta4-
> p9142SYkQ2gDH3S8PMdZg&e=).  Is there an
>
>     > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> csperkins.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=
> lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_
> tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRfcPlYZEmqGCYYi9rf9Wc7dYRz
> xspsY&e=
>
>     > >>
>
>     > >>
>
>     > >>
>
>     > >
>
>     >
>
>     >
>
>     >
>
>     > --
>
>     > Colin Perkins
>
>     > https://urldefense.proofpoint.com/v2/url?u=https-3A__
> csperkins.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=
> lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_
> tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRfcPlYZEmqGCYYi9rf9Wc7dYRz
> xspsY&e=
>
>     >
>
>     >
>
>     >
>
>
>
>
>
>

---

From: Ben Campbell <[email-masked]>
Date: Tue, 10 Oct 2017 15:59:29 -0500

> RFC 5888 Section 8.4 “FID Semantics” also states: “It is assumed that the application uses only one codec at a time to encode the media produced.  This codec MAY change dynamically during the sessi…

The argument against FID hinges on the rather philosophical question of whether the ancillary data is a truly a separate codec vs just metadata. I don’t find that particularly convincing, especially since multiple experts have weighed in on this discussion in favor of FID. Let’s say for the sake of argument that we treat is as “just metadata”. Would you argue that FID still won’t work in a practical sense?

> I could really use the help of the IESG to see how we can move forward regarding the open DISCUSS item on draft-ietf-payload-rtp-ancillary.

My takeaways from the discussion so far:

1)  LS doesn’t work, at least not cleanly.
2)  FID would work, but there are philosophical objections.
3)  People don’t want to create a new grouping semantic.

So unless people want to argue that LS really does work, even in the case of Adam’s multi-stream example, I think we are left choosing between FID or creating something new. If people don’t like FID, is creating something new (maybe a “metadata” semantic) that hard?

---

From: Adam Roach <[email-masked]>
Date: Tue, 10 Oct 2017 16:02:35 -0500

Here are the salient points that pertain to my discuss:

 1. This mechanism is broken and must not be published unless there is
    an explicit way to correlate ancillary data with its corresponding
    video stream.

 2. LS is 100% the wrong mechanism to do so.

 3. I believe that what is needed here is in the spirit of what FID is
    intended to convey, although I take Thomas' point that there is some
    language in the specific definition of FID that he believes makes it
    unsuitable for this purpose.

Given that we seem to have reached an impasse on the final point in 
particular, I would propose that the most straightforward solution here 
is to define a new group semantic (e.g., "a=group:ANC") that is 
explicitly intended to pair ancillary data with its corresponding video 
stream. This should take at most one short paragraph and an additional 
IANA registration.

/a

On 10/10/17 1:44 AM, Ali C. Begen wrote:
> Hi Thomas
>
> Please go ahead and submit a revision. Then reach out to the 
> individual DISCUSS holders to clear theirs. And then lets focus on 
> Adam's discuss.
>
> Thanks
> On Tue, Oct 10, 2017 at 3:06 AM, Thomas Edwards 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     I will note that I have uploaded a new version,
>     draft-ietf-payload-rtp-ancillary-11, to address most of the
>     comments from the IESG evaluation.
>
>     draft-ietf-payload-rtp-ancillary-11, Section 4.1 “Grouping ANC
>     Streams with other Media Streams” has two examples.
>
>     In the first SDP example, two “m=” lines are present (with each
>     having a different “c=” line), one for uncompressed video, and one
>     for ancillary data.  The two RTP sessions are identified with RFC
>     5888 Media Stream Identification Attributes (mid), and are grouped
>     together using Lip Synchronization (LS), in a fashion similar to
>     the Lip Synchronization of video and audio streams in RFC 5888
>     Section 7.1 “Example of LS”.
>
>     In the second SDP example, a single “m=” line is present without
>     any grouping, implying any synchronization would be from
>     SSRC/CNAME linkage and/or RTCP.
>
>     The most likely implementation of this RTP payload in professional
>     video production would be where multiple endpoints are
>     contributing elementary streams of media: video, audio channels or
>     channel groups, and SMPTE ST 291-1 ancillary data sources.  Those
>     endpoints (microphones, cameras, Closed Caption generators, etc.)
>     are all likely to be different, thus with different SSRCs and
>     CNAMEs, and transmitting to separate multicast IP addresses and
>     UDP ports.  However, they will likely be transmitted with a
>     commonly synchronized clock as per RFC 7273 “RTP Clock Source
>     Signalling”, most likely IEEE 1588 PTP.
>
>     The individual elementary media streams will be composed together
>     by receivers as required for production.  For example, for an
>     English language service, an English audio channel group of
>     AES67/RFC 3190 RTP audio would be composed together with the RFC
>     4175 video and the ST 291-1 ancillary data flow of English
>     language Closed Captions ancillary data and SMPTE 2016-1 AFD
>     ancillary data.    Meanwhile the Spanish language service would
>     have the same video and AFD ancillary data, but would have Spanish
>     language audio and Spanish Closed Captions.  Likely it would be a
>     distribution quality encoder (perhaps H.264 encoder) that
>     subscribes to the different streams in order to create the final
>     service.
>
>     Every time I read about “FID” in RFC 5888, I am struck by the
>     sentence in Section 8 “Flow Identification (FID)” that says “there
>     are situations where a single media instance (e.g., an audio
>     stream or a video stream) is sent using more than one RTP
>     session,” which I believe is not the case here.  For example,
>     video is one media instance, and ancillary data is a second media
>     instance.  You can’t “add” video and ancillary data together, like
>     you can add together voice audio and DTMF tone audio (as in RFC
>     5888 Section 8.2).
>
>     RFC 5888 Section 8.4 “FID Semantics” also states: “It is assumed
>     that the application uses only one codec at a time to encode the
>     media produced.  This codec MAY change dynamically during the
>     Again, this does not seem relevant to the SDP description of
>     multiple RTP streams of video, audio, and ancillary data intended
>     for synchronized presentation.  All three “codecs” are in use at
>     all times for their particular media stream.  And in particular
>     regarding draft-ietf-payload-rtp-ancillary, there is only one
>     “codec” ever used, the encapsulation of SMPTE ST 291-1 ancillary
>     data packets into RTP.
>
>     I could really use the help of the IESG to see how we can move
>     forward regarding the open DISCUSS item on
>     draft-ietf-payload-rtp-ancillary.
>
>     (I will note that SMPTE is eagerly awaiting the publishing of this
>     RFC for use as a normative reference in the upcoming ST 2110-40
>     standard, see: https://www.smpte.org/st-2110).
>
>     Thanks!
>
>     -Thomas Edwards
>      FOX
>
>     On 10/4/17, 9:27 AM, "Roni Even" <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         Hi Colin,
>
>         I agree.
>
>         There are two cases. The assumption is that both video and ANC
>     have the same source and have same CNAME (Thomas - is this correct?)
>
>
>
>         2. The video and ANC are in separate RTP sessions (two
>     m-lines). If both are a single flow (single RTP stream) should
>     group using FID. If they are not a single RTP stream (flow) should
>     use LS.
>
>
>
>         > -----Original Message-----
>
>         > From: Colin Perkins [mailto:[email-masked]
>     <mailto:[email-masked]>]
>
>         > Sent: יום ד 04 אוקטובר 2017 01:50
>
>         > To: Roni Even
>
>         > Cc: Adam Roach; Thomas Edwards; The IESG;
>     [email-masked] <mailto:[email-masked]>; draft-
>
>         > [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>
>
>         > Subject: Re: [payload] Adam Roach's Discuss on
>     draft-ietf-payload-rtp-
>
>         > ancillary-10: (with DISCUSS and COMMENT)
>
>         >
>
>         > Roni,
>
>         >
>
>         > I said nothing about multicast vs unicast, since I don’t
>     think that distinction is
>
>         > important.
>
>         >
>
>         > Sending the streams on separate m= lines or sending using a
>     single a m= line
>
>         > both work. The draft doesn’t need to discuss this, since
>     it’s standard RTP as
>
>         > would apply to any RTP session. If you use separate m=
>     lines, then you need
>
>         > a grouping mechanism. The choice of grouping mechanism is
>     what we’re
>
>         > discussing.
>
>         >
>
>         > Colin
>
>         >
>
>         >
>
>         >
>
>         >
>
>         > > On 3 Oct 2017, at 23:19, Roni Even <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         > >
>
>         > > Colin,
>
>         > > We still need some feedback from Thomas, in a previous
>     email response to
>
>         > Adam he gave a reason for using multiple RTP sessions:
>
>         > >
>
>         > > "The challenge is that the use case of the Ancillary I-D
>     within the SMPTE ST
>
>         > 2110 architecture would generally require separate network
>     destination
>
>         > addresses for the different elements (video, audio,
>     ancillary data) so they
>
>         > can be flexibly composed within broadcast plants at the
>     network level.  Your
>
>         > alternative SDP example appears to place both the video and
>     the ancillary
>
>         > data onto the same destination address
>     (https://urldefense.proofpoint.com/v2/url?u=http-3A__233.252.0.1&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=srzDHDG2LTcTBJexIDta4-p9142SYkQ2gDH3S8PMdZg&e=
>     <https://urldefense.proofpoint.com/v2/url?u=http-3A__233.252.0.1&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=srzDHDG2LTcTBJexIDta4-p9142SYkQ2gDH3S8PMdZg&e=>).
>     Is there an
>
>         > appropriate SDP mechanism to communicate the intimate
>     relationship
>
>         > between the RTP streams, but also have them transmitted on
>     separate
>
>         > destination IP addresses (and synchronized)?
>
>         > >
>
>         > > (In production practice, this may not be a significant
>     issue as the RTP
>
>         > streams to be combined by IP receivers for conversion to
>     SDI/HDMI or for
>
>         > input to a distribution encoder may be specified by a
>     control system directly
>
>         > to the receiver using an API that isn’t just an SDP file.)"
>
>         > >
>
>         > > I want to assume that by different destination addresses
>     it refers to
>
>         > multicast addresses like above and not to different unicast
>     IP addresses.
>
>         > > You are right that for unicast it will make sense to use
>     the same RTP session
>
>         > with the same SSRC.
>
>         > >
>
>         > > Roni
>
>         > >
>
>         > >> -----Original Message-----
>
>         > >> From: Colin Perkins [mailto:[email-masked]
>     <mailto:[email-masked]>]
>
>         > >> Sent: יום ד 04 אוקטובר 2017 00:46
>
>         > >> To: Roni Even
>
>         > >> Cc: Adam Roach; Thomas Edwards; The IESG;
>     [email-masked] <mailto:[email-masked]>;
>
>         > >> draft- [email-masked]
>     <mailto:[email-masked]>; [email-masked]
>     <mailto:[email-masked]>;
>
>         > >> [email-masked] <mailto:[email-masked]>
>
>         > >> Subject: Re: [payload] Adam Roach's Discuss on
>
>         > >> draft-ietf-payload-rtp-
>
>         > >> ancillary-10: (with DISCUSS and COMMENT)
>
>         > >>
>
>         > >>
>
>         > >>> On 3 Oct 2017, at 21:40, Roni Even <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         > >>>
>
>         > >>> Hi Colin,
>
>         > >>> I re-read RFC5888 and tried to understand what is the
>     difference
>
>         > >>> between FID and LS I was trying to understand what is a
>     flow and
>
>         > >>> from section 8.3
>
>         > >>>
>
>         > >>> "The previous examples show that the definition of a
>     media stream in
>
>         > >>> [RFC2326] does not cover some scenarios.  It cannot be
>     assumed that
>
>         > >>> a  single media instance maps into a single RTP
>     session.  Therefore,
>
>         > >>> we  introduce the definition of a media flow:
>
>         > >>>
>
>         > >>>     A media flow consists of a single media instance,
>     e.g., an audio
>
>         > >>>     stream or a video stream as well as a single
>     whiteboard or shared
>
>         > >>>     application group.  When using RTP, a media flow
>     comprises one or
>
>         > >>>     more RTP sessions."
>
>         > >>>
>
>         > >>> It looks to me that a media flow is a SINGLE media
>     stream that spans
>
>         > >>> over
>
>         > >> multiple RTP sessions (the term media instance is a what
>     we call a
>
>         > >> media stream in the taxonomy). Section 8.5 (8.5.1)
>     enhance this by
>
>         > >> saying when FID cannot be used " FID semantics are useful
>     when the
>
>         > application only uses
>
>         > >> one codec at   a time"  which again describe a flow as a
>     single media
>
>         > stream.
>
>         > >>>
>
>         > >>> I am not sure how ANC and the video relates to each
>     other this is
>
>         > >>> what I am
>
>         > >> asking Thomas to be more clear. If the video and ANC are
>     one stream
>
>         > >> over multiple RTP sessions FID is correct otherwise I
>     think the LS should be
>
>         > used.
>
>         > >>
>
>         > >> My understanding is that the video and ancillary data are
>     the same
>
>         > >> stream. In broadcast TV the ancillary data is sent
>     in-band, in the blanking
>
>         > intervals.
>
>         > >>
>
>         > >>> My reading on  the ancillary draft is that ANC and the
>     video are
>
>         > >>> what we call
>
>         > >> a single RTP stream but this is not what Thomas says.
>
>         > >>
>
>         > >>
>
>         > >> I expect one could send the ancillary data using the same
>     SSRC, and
>
>         > >> the same 5-tuple, as the media data (in the same way
>     comfort noise
>
>         > >> and speech data can be sent), in which case you don’t
>     need the
>
>         > >> grouping. If they’re sent as separate RTP streams, with
>     separate m=
>
>         > >> lines, then you need grouping. I believe FID makes sense
>     for this.
>
>         > >>
>
>         > >> --
>
>         > >> Colin Perkins
>
>         > >>
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__csperkins.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRfcPlYZEmqGCYYi9rf9Wc7dYRzxspsY&e=>
>
>         > >>
>
>         > >>
>
>         > >>
>
>         > >
>
>         >
>
>         >
>
>         >
>
>         > --
>
>         > Colin Perkins
>
>         >
>     <https://urldefense.proofpoint.com/v2/url?u=https-3A__csperkins.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRfcPlYZEmqGCYYi9rf9Wc7dYRzxspsY&e=>
>
>         >
>
>         >
>
>         >
>
>
>
>
>
>

---

From: "Ali C. Begen" <[email-masked]>
Date: Wed, 11 Oct 2017 00:40:01 +0300

Hi Ben

On Tue, Oct 10, 2017 at 11:59 PM, Ben Campbell <[email-masked]> wrote:

> So unless people want to argue that LS really does work, even in the case
> of Adam’s multi-stream example, I think we are left choosing between FID or
> creating something new. If people don’t like FID, is creating something new
> (maybe a “metadata” semantic) that hard?

Yes, this is where we are. LS does not work, I agree. I believe FID works
as the language in the RFC does not bother me because whoever needs to work
with ANC data knows how to deal with it anyway. So bottomline is I am ready
to ship this with FID. If that is absolutely not acceptable to Thomas or
others, the only option is to define a new grouping semantics, which is not
a significant work but needs blessing from MMUSIC.

-acbegen

---

From: "Ali C. Begen" <[email-masked]>
Date: Wed, 11 Oct 2017 00:41:20 +0300

On Wed, Oct 11, 2017 at 12:02 AM, Adam Roach <[email-masked]> wrote:

>    an explicit way to correlate ancillary data with its corresponding video
>    stream.
>
> particular, I would propose that the most straightforward solution here is
> to define a new group semantic (e.g., "a=group:ANC") that is explicitly
> intended to pair ancillary data with its corresponding video stream. This
> should take at most one short paragraph and an additional IANA registration.
>

Ah, did not see your email before my earlier email. I'd agree but I would
do the new grouping semantics for generic metadata not just for ANC.

-acbegen

>> draft-ietf-payload-rtp-ancillary-11, to address most of the comments
>> from the IESG evaluation.
>>
>> (I will note that SMPTE is eagerly awaiting the publishing of this RFC
>> for use as a normative reference in the upcoming ST 2110-40 standard, see:
>>     > data onto the same destination address (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__233.
>> 252.0.1&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-
>> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=
>> dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=
>> srzDHDG2LTcTBJexIDta4-p9142SYkQ2gDH3S8PMdZg&e=).  Is there an
>>
>>     > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__csperki
>> ns.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-
>> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=
>> dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRf
>> cPlYZEmqGCYYi9rf9Wc7dYRzxspsY&e=
>>
>>     > >>
>>
>>     > >>
>>
>>     > >>
>>
>>     > >
>>
>>     >
>>
>>     >
>>
>>     >
>>
>>     > --
>>
>>     > Colin Perkins
>>
>>     > https://urldefense.proofpoint.com/v2/url?u=https-3A__csperki
>> ns.org_&d=DwIGaQ&c=uw6TLu4hwhHdiGJOgwcWD4AjKQx6zvFcGEsbfiY9-
>> EI&r=lekNOOM5noV61zrPH3rwPyhtNnLLWoLEHgd0quQxly8&m=
>> dOxHA7dn5d2eo_tU3divd4ks6mmXPhk5WmodLULHUPY&s=s2We2es-My1JRf
>> cPlYZEmqGCYYi9rf9Wc7dYRzxspsY&e=
>>
>>     >
>>
>>     >
>>
>>     >
>>
>>
>>
>>
>>
>>
>
>

---

From: Ben Campbell <[email-masked]>
Date: Tue, 10 Oct 2017 16:56:23 -0500

> On Oct 10, 2017, at 4:40 PM, Ali C. Begen <[email-masked]> wrote:
> 
> Hi Ben
> 
> Yes, this is where we are. LS does not work, I agree. I believe FID works as the language in the RFC does not bother me because whoever needs to work with ANC data knows how to deal with it anyway. So bottomline is I am ready to ship this with FID. If that is absolutely not acceptable to Thomas or others, the only option is to define a new grouping semantics, which is not a significant work but needs blessing from MMUSIC.

The IANA registration policy for the group semantics table is “standards action”. It needs a standards track RFC, but I am not aware of a requirement for that to come from MMUSIC. The most recent one came from AVTEXT. I agree that we would want a new semantic to be at least reviewed in MMUSIC. But I don’t think we are talking about all that much text to add.

FID would still be easier, though.  :-)

Ben.

---

From: Adam Roach <[email-masked]>
Date: Tue, 10 Oct 2017 17:01:04 -0500

> FID would still be easier, though.  :-)

To be clear, I absolutely agree that FID would be cleaner.

/a

---

From: Adam Roach <[email-masked]>
Date: Tue, 10 Oct 2017 17:04:55 -0500

On 10/10/17 4:41 PM, Ali C. Begen wrote:
>
> On Wed, Oct 11, 2017 at 12:02 AM, Adam Roach <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     Given that we seem to have reached an impasse on the final point
>     in particular, I would propose that the most straightforward
>     solution here is to define a new group semantic (e.g.,
>     "a=group:ANC") that is explicitly intended to pair ancillary data
>     with its corresponding video stream. This should take at most one
>     short paragraph and an additional IANA registration.
>
>
> Ah, did not see your email before my earlier email. I'd agree but I 
> would do the new grouping semantics for generic metadata not just for 
> ANC.

I'm okay either way.

Some obeservations: If we do a new type just for ANC, I think we could 
include it as part of this document. If we do it as a generic "metadata" 
type, I think we'd need to have a separate document for that. I imagine 
the separate document (and the more generic requirements) would take 
more time than adding to this document; and it would be a normative 
dependency for draft-ietf-payload-rtp-ancillary.

But, again, I'm okay with either such approach.

/a

---

From: "Ali C. Begen" <[email-masked]>
Date: Wed, 11 Oct 2017 01:09:08 +0300

On Wed, Oct 11, 2017 at 1:04 AM, Adam Roach <[email-masked]> wrote:

> type, I think we'd need to have a separate document for that. I imagine the
> separate document (and the more generic requirements) would take more time
> than adding to this document; and it would be a normative dependency for

Right, so I suggest we use FID for this draft and ship it. And once we have
a volunteer, he can propose a grouping semantics for generic metadata in
MMUSIC which will hopefully help future drafts similar to this one. @Ben,
grouping semantics does not need to come from MMUSIC WG but that is where
most people familiar with the issues are, hence my comment on needing
MMUSIC's blessing.

@Thomas, what do you think?

-acbegen

---

From: Thomas Edwards <[email-masked]>
Date: Wed, 11 Oct 2017 00:32:30 +0000

After re-reading Adam Roach’s DISCUSS item again, I think I now comprehend his concern.  I think he is saying that there is a need to show a “relationship” between an ancillary data stream and a video stream, not just a “synchronization” (although that is required as well).

In an SDP of just one video stream and one ancillary stream, I believe that it is unambiguous that the single ancillary data is “related” to the single video.

But clearly in an SDP of multiple video streams and multiple ancillary streams, even if you wanted them all synchronized for presentation, there is no information about which ancillary stream is related to which video stream.

The “relationship” does matter for many ancillary data items.  For example, the Active Format Description (AFD) relates specifically to the appropriate conversion of material in a particular video stream from 16:9 pictures to 4:3 displays, and can change on a frame-by-frame basis.  You also might want Closed Captions showing up on the same display as the video it is related to.

Unfortunately, while FID could suggest this “relationship”, RFC 5888 says nothing about FID implying “synchronized presentation”.  I am very concerned about being clear that the grouping should imply synchronized presentation where appropriate.

The good news is that RFC 5888 states: “There MAY be several "a=group" lines in a session description.”

So I’d like to suggest this for draft-ietf-payload-rtp-ancillary section 4.1. “Grouping ANC Streams with other Media Streams”:

1) Provide an example with a single video RTP stream and a single ancillary data RTP stream, grouped with “LS”, and note that in such a situation, the relationship between video and ancillary RTP stream is unambiguous.  (I suspect this would be the use case for 90% of professional media implementations)

2) Provide a second example with two video RTP streams and two ancillary data RTP streams, grouped with “LS” and also grouped with “FID” to show the relationship between video and ancillary RTP streams

That is:
a=group:LS V1 V2 M1 M2
a=group:FID V1 M1
a=group:FID V2 M2

(where V1=video #1, V2=video #2, M1=ancillary #1, M2=ancillary #2).

And specifically calling out that “FID” is used to show that an ancillary RTP stream is to be associated with a video RTP stream.  (I can see this use case for multiviewers in professional media applications – for example, there are 50 multiviewers of 4 video streams each on the Game Creek Yogi IP outside broadcast truck’s production monitor wall that will be used to produce the Major League Baseball World Series on FOX later this month).

Let me know if this sounds like a good compromise!

-Thomas

From: "Ali C. Begen" <[email-masked]>
Date: Tuesday, October 10, 2017 at 3:09 PM
To: Adam Roach <[email-masked]>
Cc: Thomas Edwards <[email-masked]>, Roni Even <[email-masked]>, Colin Perkins <[email-masked]>, The IESG <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [payload] Adam Roach's Discuss on draft-ietf-payload-rtp-ancillary-10: (with DISCUSS and COMMENT)

On Wed, Oct 11, 2017 at 1:04 AM, Adam Roach <[email-masked]<mailto:[email-masked]>> wrote:
On 10/10/17 4:41 PM, Ali C. Begen wrote:

On Wed, Oct 11, 2017 at 12:02 AM, Adam Roach <[email-masked]<mailto:[email-masked]>> wrote:

Given that we seem to have reached an impasse on the final point in particular, I would propose that the most straightforward solution here is to define a new group semantic (e.g., "a=group:ANC") that is explicitly intended to pair ancillary data with its corresponding video stream. This should take at most one short paragraph and an additional IANA registration.

Ah, did not see your email before my earlier email. I'd agree but I would do the new grouping semantics for generic metadata not just for ANC.

I'm okay either way.

Some obeservations: If we do a new type just for ANC, I think we could include it as part of this document. If we do it as a generic "metadata" type, I think we'd need to have a separate document for that. I imagine the separate document (and the more generic requirements) would take more time than adding to this document; and it would be a normative dependency for draft-ietf-payload-rtp-ancillary.

Right, so I suggest we use FID for this draft and ship it. And once we have a volunteer, he can propose a grouping semantics for generic metadata in MMUSIC which will hopefully help future drafts similar to this one. @Ben, grouping semantics does not need to come from MMUSIC WG but that is where most people familiar with the issues are, hence my comment on needing MMUSIC's blessing.

-acbegen

/a

---

From: Thomas Edwards <[email-masked]>
Date: Wed, 18 Oct 2017 00:34:51 +0000

After further discussion with Adam Roach, I am now convinced that “FID” grouping is appropriate.  Thus, I propose that Section 4.1 of the I-D be altered to state:

4.1 Grouping ANC Streams with other Media Streams

To indicate the association of an ancillary data stream with a particular video stream, implementers MAY group the "m" lines together using Flow Identificaiton ("FID") semantics as defined in RFC 5888 [RFC5888].

A sample SDP mapping for grouping ANC data with RFC 4175 video is as follows:

      v=0
      o=Al 123456 11 IN IP4 host.example.com
      c=IN IP4 233.252.0.1/255
      c=IN IP4 233.252.0.2/255

-Thomas

---

From: "Ali C. Begen" <[email-masked]>
Date: Wed, 18 Oct 2017 09:59:47 +0300

Sounds good, I am glad we have an agreement.

-acbegen

> On Oct 18, 2017, at 3:34 AM, Thomas Edwards <[email-masked]> wrote:
>