Subject: [Cellar] Status of the ChannelPositions element

From: Tobias Rapp <[email-masked]>
Date: Fri, 8 Sep 2017 14:14:08 +0200

Hi,

when looking for a way to store audio channel positions in Matroska I
stumbled over the "ChannelPositions" element. Currently it is defined as
having a binary value but unfortunately the specification doesn't
actually mention the format (see section 7.3.145).

If I remember correctly, Steve Lhomme mentioned some time ago that this 
element was once defined but never used. So I guess the (current)
ChannelPositions element definition can be removed from the specs?

What would be a suitable replacement? I think it should at least support 
the typical named channel positions like 
(Front)Left/(Front)Right/Center/LFE/..., possibly using a list similar 
to table 7 in section "Speaker Channel Position" of ISO/IEC 23001-8:2013 
for mapping positions to numerical values (maybe with an "Unknown" 
position value inserted at the front).

Likely 3D positions will be important in the future. Does anybody have 
some experience in how 3D positions are described (Google/WebM 
developers)? At least I'd suggest to use a "type" flag that indicates 
whether the position list contains named positions, 3D coordinates or 
some other future data format.

Or we take the same approach as used for the Projection elements and 
simply adopt the data structure of ISOBMFF box "chnl". The pros and cons 
in my opinion:

(+) allows for more simple re-wrapping between MP4 and MKV;
(-) adds MP4 container semantics to MKV (FullBox flags, 
alignment/endianness requirements)
(-) document version available to me (ISO/IEC 14496-12:2015E) doesn't 
really describe what an "object" is and how it is stored;
(-) due to using the speaker position list of ISO/IEC 23001-8 doesn't 
allow some of the channels to indicate unknown position;

In the end my proposal would be to not directly follow ISOBMFF. Instead 
we could do:

(a) add element "ChannelPositionType" of type enum/integer with values:

1 => Channel Configuration (based on ISO/IEC 23001-8:2013, section 
"Channel Configuration")
2 => Channel Mask (based on WAVEFORMATEXTENSIBLE, field "dwChannelMask")
3 => Position List (based on ISO/IEC 23001-8:2013, section "Speaker 
Channel Position")
4 => 3D Position List (based on ?)

(b) add element "ChannelPositionData/ChannelPositionPrivate" with data 
type depending on the value of ChannelPositionType:

1 => unsigned integer
2 => unsigned integer
3 => array of unsigned integers, length is defined by "Channels" 
element, 0=Unknown, 1=Left, 2=Right, ...
4 => array of speaker coordinate structure (TBD), length is defined by 
"Channels" element

Suggestions and opinions welcome.

Regards,
Tobias

---

From: Michael Bradshaw <[email-masked]>
Date: Fri, 8 Sep 2017 10:17:02 -0700

On Fri, Sep 8, 2017 at 5:14 AM, Tobias Rapp <[email-masked]> wrote:
> some experience in how 3D positions are described (Google/WebM developers)?

YouTube generally uses Opus with channel mapping families defined by:
https://tools.ietf.org/html/draft-ietf-codec-ambisonics
That is, the decoded audio is ambisonic spatial audio with SN3D
normalization in ACN channel order. There are others, though. Dolby's Atmos
stuff uses actual object positions for each sound (and they move around;
they're not fixed). There are others I've heard of but can't remember
enough details to coherently talk about them. In short, there are several
very different ways people have historically represented 3D audio (and that
has persisted to the present day, and will continue for the foreseeable
future IMO).

The Opus spatial audio we're using gets its channel mapping stored in the
OpusHead packet (which is stored in Matroska's CodecPrivate element). So
unfortunately this work is complicated by the fact that some audio codecs
already define channel mappings/positions (and don't need more help from
Matroska), but others don't (e.g., raw LPCM) and need help from Matroska to
specify what each channel means.

In the end, if Matroska does add something like this, I think something
along the lines of option (a) is more appealing, though I'm not a big fan
of multiple ways to represent the same information. That just adds
complexity to parsers/players (I'd personally rather put the complexity in
the muxer/encoder, since I think there are fewer muxers in existence and
muxing is less common a task than parsing/playing).

--Michael

---

From: Tobias Rapp <[email-masked]>
Date: Mon, 11 Sep 2017 11:19:04 +0200

Hi Michael,

thanks for your informative summary about handling ambisonics in Ogg/Opus.

> The Opus spatial audio we're using gets its channel mapping stored in  > the OpusHead packet (which is stored in Matroska's CodecPrivate > 
element). So unfortunately this work is complicated by the fact that > 
some audio codecs already define channel mappings/positions (and don't > 
need more help from Matroska), but others don't (e.g., raw LPCM) and > 
need help from Matroska to specify what each channel means.
Ok, so there should be some statement that in case no channel mapping 
elements are defined on container level the audio channel mapping should 
be taken from the codec level. Is it necessary to explicitly disallow 
storing the mapping on both container _and_ codec level to avoid muxer 
conflicts?

> along the lines of option (a) is more appealing, though I'm not a big 
> fan of multiple ways to represent the same information. That just adds 
> complexity to parsers/players (I'd personally rather put the complexity 
> in the muxer/encoder, since I think there are fewer muxers in existence 
> and muxing is less common a task than parsing/playing).

My idea was to implement both elements (a) and (b) together, one 
indicating the type and the other containing the actual data. Both 
elements could be merged into one, using the first byte(s) as a flag for 
the type of data that is following. I had chosen two separate elements 
as it seemed to better match the EBML-style, but I have no personal 
preference.

And yes, not all proposed ChannelPositionType values are actually 
necessary. Having worked a lot with BWF/WAVE files my minimal 
requirement would be the "Channel Mask" value type.

Regards,
Tobias

---

From: Tobias Rapp <[email-masked]>
Date: Fri, 22 Sep 2017 14:59:22 +0200

Hi,

there are some possible directions for further development of this topic 
and I'd like to get some feedback on which one is preferred:

Option 1: Don't add anything regarding audio channel positions to the 
Matroska spec. Information is available to players for audio codecs that 
include channel position data as part of the encoded bitstream (Opus, 
AAC), for audio codecs that miss this information the player has to make 
assumptions (PCM).

Option 2: Add some generic audio channel position element(s) to the 
Matroska spec. Information is available to players for all audio codecs, 
in case the encoded bitstream also contains channel position data the 
Matroska spec should define which one takes precedence.

Option 3: Add audio channel position data to the codec mapping section 
of the Matroska spec. Channel position data is stored within 
CodecPrivate element only for audio codecs that do not provide the 
information as part of the bitstream (PCM).

Opinions?

Best regards,
Tobias

---

From: Reto Kromer <[email-masked]>
Date: Fri, 22 Sep 2017 15:24:48 +0200

Tobias Rapp wrote:

>there are some possible directions for further development of
>this topic and I'd like to get some feedback on which one is
>preferred:

Ha! Will you be in Vienna? I have included in my wish list that
one too. In my presentation, if accepted, I have included some
wishes for all - EBML, Matroska, FFV1 and FLAC -, originated on
my experience from the field.

Best regards, Reto

---

From: Tobias Rapp <[email-masked]>
Date: Fri, 22 Sep 2017 16:40:27 +0200

> my experience from the field.

No, but some of my Vienna colleagues will be there. Is some 
Matroska/FFV1 working meeting planned for No Time To Wait 2?

Best regards,
Tobias

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 24 Sep 2017 15:49:54 +0200

2017-09-22 14:59 GMT+02:00 Tobias Rapp <[email-masked]>:
> Hi,
>
> there are some possible directions for further development of this topic and
> I'd like to get some feedback on which one is preferred:
>
> include channel position data as part of the encoded bitstream (Opus, AAC),
> for audio codecs that miss this information the player has to make
> assumptions (PCM).
>
> Option 2: Add some generic audio channel position element(s) to the Matroska
> spec. Information is available to players for all audio codecs, in case the
> encoded bitstream also contains channel position data the Matroska spec
> should define which one takes precedence.
>
> Option 3: Add audio channel position data to the codec mapping section of
> the Matroska spec. Channel position data is stored within CodecPrivate
> element only for audio codecs that do not provide the information as part of
> the bitstream (PCM).

There might be option 4: always have a value set in the container and
optionally in the codec if it wants to. One of the possible value(s)
in Matroska could be a special value "see codec" for things that are
too complex for Matroska.

In general, for playback, it's good to know before you start decoding
how you have to configure the audio output and that involves how many
channels and how they interact with each other. And the best way for
that is to look at what the container says. Before parsing the actual
data and opening the decoder. With PCM we need to have as many
mappings as possible anyway. It can't be left to interpretation.

I like the 2 elements approach. For stereo, multichannel and
object-based things can be quite different. It's at least good to know
if a track is either 3 of these types (there might be more) before
selecting it. And the way we store the actual mapping for each can be
different. But we need a way to specify how each format is going to be
stored. We may even have 3 elements: multichannel, ac-3 style, binary
blob with a custom mapping.

> Opinions?
>
> Best regards,
>
> Tobias
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Tobias Rapp <[email-masked]>
Date: Mon, 25 Sep 2017 17:24:13 +0200

> too complex for Matroska.

Looks like a variant of option (2) data-location wise (i.e. stored in 
generic elements, not codec private data). Likely the "see codec" value 
would be used as a default for backwards compatibility.

> blob with a custom mapping.

Not sure how channel mapping is done for AC-3. Do you refer to the 
"Audio Coding Mode"?

So it seems we agree that using a "ChannelPositionType" / 
"ChannelPositionData" element pair would be the most flexible option. 
What would be the purpose of the third element you mentioned?

Regards,
Tobias

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 25 Sep 2017 17:33:06 +0200

2017-09-25 17:24 GMT+02:00 Tobias Rapp <[email-masked]>:
>>> and
>>> AAC),
>>> Matroska

Yes

> Not sure how channel mapping is done for AC-3. Do you refer to the "Audio
> Coding Mode"?
>
> "ChannelPositionData" element pair would be the most flexible option. What
> would be the purpose of the third element you mentioned?

For arbitrary values. For example you have just top and bottom
channels or north/south/east/west. I don't know if there are existing
ways to define all kinds of channel formats, or even some formats that
are kind of generic and we can reuse.