Subject: Re: [Cellar] TrackTypes for non-standard data formats

From: Steve Lhomme <[email-masked]>
Date: Wed, 26 Apr 2017 10:47:08 +0200

2017-02-27 3:21 GMT+01:00 Matt Gruenke <[email-masked]>:
> Where are the complex & ancillary TrackTypes described?  If ancillary tracks

'complex' is track type 3:
https://github.com/Matroska-Org/matroska-specification/blob/master/ebml_matroska.xml#L198
Ancillary is not defined yet. IMO it's less generic/custom than Complex.

> can be named, that would probably be sufficient.  I think the CodecID should
> describe only how the data is encoded, which might be insufficient to convey
> its meaning or relationship to the other tracks.

There's 'TrackOverlay' that defined one type of relation between
tracks but that's it. I think it would be hard to describe all kinds
relations that can exist between custom (non standard) tracks and
other custom tracks.

> In all of the examples I mentioned, I can foresee cases where these would
> need to be time-stamped frames that might have a different sampling rate
> than the video or audio streams, if present.  In fact, this is the case for
> one camera I've used, where the depth sensor operates at a different rate
> than the primary image sensor.

There is no 'sampling rate' in Matroska. Only the timestamp of a Block
matters. So audio/video/other don't need to share anything in that
regard.

>
> Matt
>
>
> On Sun, Feb 26, 2017 at 11:24 AM, Steve Lhomme <[email-masked]> wrote:
>>
>> In general when there's something you want to multiplex there's always
>> the "complex" type in which you may put whatever you want.
>>
>> But it is recommended to use existing things or create what's missing.
>>
>> 2017-01-25 5:39 GMT+01:00 Matt Gruenke <[email-masked]>:
>> > If it'd help to deal with specifics, some examples I have in mind are
>> > scene
>> > geometry (e.g. point clouds, polygonal meshes, Z-buffer), camera pose
>> > (e.g.
>> > position & orientation as vector & quaternion pairs, GPS coordinates,
>> > compass readings), and various telemetry data (plenty of examples at
>> > https://en.wikipedia.org/wiki/Telemetry ).
>>
>> For things like camera position/GPS, if it doesn't change, a tag on a
>> chapter for that specific thing is probably better. It's easier to
>> lookup such information among files (theoretically). But obviously
>> there are lots of cases where this is not the case and we probably
>> need a kind of track specific for that (sport cameras like Go Pros
>> come to mind) with proper definition. Maybe the ancillary name is
>> generic enough to cover all these and then depending on the codec you
>> know what kind of information you would get.
>>
>> For the scene geometry I'm not familiar with this. Does it need to be
>> interleaved with the audio/video things ? A chapter tag is not enough
>> ?
>>
>> > Although nothing specific comes to mind that wouldn't have a stream-like
>> > structure, one might allow for more such things as menus and button
>> > tracks.
>> > Perhaps a game streaming application might use a track for player
>> > information and statistics.
>> >
>> > In considering the utility & possible necessity of multiple,
>> > application-defined TrackTypes, perhaps it's worth reviewing the
>> > purposes
>> > they serve.  One function of TrackType is certainly to simplify semantic
>> > mapping of the available tracks.  To this end, it can prove more
>> > reliable
>> > than CodecID, since it's certainly possible to use the same encoding for
>> > tracks with different semantics.
>> >
>> > Of course, TrackType also dictates the content model of TrackEntry.
>> > Though,
>> > I don't know enough about EBML to have a sense of whether it's practical
>> > and
>> > reasonable for applications to extend the schema to allow for custom
>> > track
>> > metadata.
>> >
>> > I guess I'd ask why *not* allow ancillary data to have one of ~8
>> > TrackTypes?
>> > At the very least, I'd suggest using somewhere near the top of the
>> > range.
>> > Then, more values for ancillary data could be added, subsequently, while
>> > keeping the range contiguous.
>> >
>> >
>> > Matt
>> >
>> >
>> > On Tue, Jan 24, 2017 at 10:24 AM, Dave Rice <[email-masked]> wrote:
>> >>
>> >>
>> >> IIRC the proposal for ancillary tracks was to have Codec_IDs prefixed
>> >> with
>> >> "N_". If your binary data stream is wholly private with contents that
>> >> are
>> >> not intended to be known by the Matroska specification then perhaps a
>> >> private Codec ID prefix (such a "X_") would be worthwhile to define. On
>> >> the
>> >> other hand, if you could be more transparent you could also consider
>> >> defining your codec openly, perhaps the application specific use could
>> >> have
>> >> wider interest.
>> >>
>> >> Or could OggKate store the values you need? Perhaps it's worthwhile to
>> >> define storage for a time-based metadata track such as this.
>> >>
>> >> Dave Rice
>> >
>> >
>> >
>> > _______________________________________________
>> > Cellar mailing list
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/cellar
>> >
>>
>>
>>
>> --
>> Steve Lhomme
>> Matroska association Chairman
>
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Matt Gruenke <[email-masked]>
Date: Thu, 27 Apr 2017 22:53:19 -0400

> matters. So audio/video/other don't need to share anything in that

What I meant by 'sampling rate' wasn't that there would be some exact frame
rate.  I simply meant that there might not be a 1-to-1 relationship between
frames of different types.  Again, to use the example of a certain camera
with a depth sensor, it can capture RGB frames at a different rate than
depth frames.

I mention this simply to point out that the depth frames seem like they
belong in their own track.

Matt

---

From: Dave Rice <[email-masked]>
Date: Fri, 28 Apr 2017 20:37:27 -0400

> On Apr 27, 2017, at 10:53 PM, Matt Gruenke <[email-masked]> wrote:
> 
> On Wed, Apr 26, 2017 at 4:47 AM, Steve Lhomme <[email-masked] <mailto:[email-masked]>> wrote:
> 2017-02-27 3:21 GMT+01:00 Matt Gruenke <[email-masked] <mailto:[email-masked]>>:
>  
> What I meant by 'sampling rate' wasn't that there would be some exact frame rate.  I simply meant that there might not be a 1-to-1 relationship between frames of different types.  Again, to use the example of a certain camera with a depth sensor, it can capture RGB frames at a different rate than depth frames.
> 
> I mention this simply to point out that the depth frames seem like they belong in their own track.

I regards to this discussion, I ended up reviewing https://tools.ietf.org/html/rfc6648 <https://tools.ietf.org/html/rfc6648> which covers the deprecation of the "X-" prefix as a method to distinguish standardized and unstandardized descriptions. In particular I think the recommendations here https://tools.ietf.org/html/rfc6648#section-3 <https://tools.ietf.org/html/rfc6648#section-3> make sense to follow in the use of Codec IDs, in that a developer of an unstandardized codec id should do so with an assumption that that Codec ID may become standardized.

I sent a pull request for review at https://github.com/Matroska-Org/matroska-specification/pull/117 <https://github.com/Matroska-Org/matroska-specification/pull/117>, which adds this to the Codec Mapping page.

## Recommendations for the Creation of New Codec Mappings

Creators of new Codec Mappins to be used in the context of Matroska:

- SHOULD assume that all Codec Mappings they create might become standardized, public, commonly deployed, or usable across multiple implementations.

- SHOULD employ meaningful values for Codec ID and Codec Name that they have reason to believe are currently unused.

- SHOULD NOT prefix their Codec ID with "X-" or similar constructs.

These recommendations are based upon Section 3 of [@!RFC6648].

Best Regards,
Dave Rice

---

From: Matt Gruenke <[email-masked]>
Date: Sat, 29 Apr 2017 16:20:38 -0400

On Fri, Apr 28, 2017 at 8:37 PM, Dave Rice <[email-masked]> wrote:

> I sent a pull request for review at https://github.com/
> Matroska-Org/matroska-specification/pull/117, which adds this to the
> Codec Mapping page.
>
> - SHOULD assume that all Codec Mappings they create might become
> standardized, public, commonly deployed, or usable across multiple
> implementations.
>
> - SHOULD employ meaningful values for Codec ID and Codec Name that they
> have reason to believe are currently unused.
>

It's important to note that while IANA deprecated the X- prefix, they *replaced
*it with the Vendor and Personal trees.

https://tools.ietf.org/html/rfc6838#section-3

So, you can certainly say "don't use X_".  But, if someone has an encoding
that doesn't fit any of the existing prefixes, then using "Y_" is just as
bad (or worse).

Matt