Subject: [Cellar] CodecIDs for non-standard and unregistered data formats

From: Matt Gruenke <[email-masked]>
Date: Tue, 14 Feb 2017 01:46:52 -0500

Related to my recent query about TrackTypes for non-standard data formats,
I'd like to discuss CodecIDs.  I had previously raised this issue, on the
matroska-devel list, and was advised to take it up, here.  In the interim,
my thinking on the matter has evolved.

I wonder why not allow IANA Media Types (aka MIME types), as a sub-class of
CodecIDs?  Of course, the official Matroska CodecID would be preferred, for
all formats for which one has been designated.  However, this would enable
broader use of Matroska files, including for private, application-specific
data stream formats.

Perhaps a prefix like 'X_MEDIATYPE/' could simply be prepended to media
type names conforming with RFC 6838.  The value of TrackType would still
determine whether video, audio, or any other type-specific elements can be
used in such tracks.

Thank you for considering my suggestion.

Matt

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 26 Feb 2017 17:29:35 +0100

> broader use of Matroska files, including for private, application-specific

The main goal of Matroska is multimedia interleaved/timestamped data.
It's possible to have "complex" track and put in it whatever you want.
If a timestamp is required. If it's not, using attachment is what you
want and it already has a MIME type.

> Perhaps a prefix like 'X_MEDIATYPE/' could simply be prepended to media type
> names conforming with RFC 6838.  The value of TrackType would still

Although tempting I think it's better to avoid interleaving all kinds
of stuff with audio/video. You could want to embed a "video/quicktime"
but it would not be interleaved, or you'd need to parse/mux the data
and then it's not a clean "video/quicktime" you get anymore.

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 26 Feb 2017 17:59:41 +0100

> and then it's not a clean "video/quicktime" you get anymore.

I don't think this is the point.
Let's imagine we don't have any AVI compatibility layer (for whatever 
reason, e.g. lawyers convinced a country that BITMAPINFOHEADER / 
WAVEFORMATEX structure are patented), we have no other way for having 
e.g. AMR, G719, G729 and so on because it is not defined in native 
Matroska codec identifiers, but IANA media types have them (so a muxer 
could use "X_MEDIATYPE/audio/G729").
Consider Matt suggestion as similar to the AVI compatibility layer: this 
is just another way to define a codec identifier, also for 
interleaved/timestamped video and audio content.

Right now, I am thinking to TTML fragments (subtitles), instead of 
defining any TTML stuff in Matroska specs, 
"X_MEDIATYPE/application/ttml+xml" could be used and we rely on 
https://www.iana.org/assignments/media-types/application/ttml+xml .
For crazy ideas but we should not forbid crazy ideas :), someone may 
want to use PDF as a video format with one PDF page per video frame 
(similar to MJPEG), and instead of defining and requesting a "V_PDF" 
codec identifier, one would use "X_MEDIATYPE/application/pdf".

At long term, a potential decision would be to delegate codec 
identifiers management to IANA as they already do the job (for RTP and 
so on) in order to not duplicate efforts, and use Matroska list only 
when IANA does not fit Matroska needs, i.e. why should we care of having 
a spec about "V_MPEG4/ISO/AVC" (note: it is not on the codec spec page 
but a lot used) if we could rely on "video/H264" registration? (this is 
theoretical, "V_MPEG4/ISO/AVC" is already used. but we could do that for 
upcoming AV1 defined by IETF).

Jérôme

---

From: Matt Gruenke <[email-masked]>
Date: Sun, 26 Feb 2017 22:04:56 -0500

Let's take this example of a camera having both a visible light sensor and
a depth sensor, as well as a stereo microphone.  For various reasons, its
depth sensor captures frames at different times and a different rate than
its color image sensor.  I'd like to record a video file containing an
audio track, a normal video track, and a depth track.  I'd like the audio
and video portions to be playable in normal video player apps.  When using
enhanced apps, the depth data may also be utilized.

Today, most players I've tried will behave as I've described, if I simply
write the depth data using an unspecified TrackType and having a custom
CodecID.  However, not all players handle it gracefully (or at all), and
there's no guarantee that the others will continue to work.

Having a dedicated TrackType (or range, thereof) would solve one potential
problem.  Perhaps the Ancillary TrackType already meets this need.

Using nonstandard CodecIDs is another concern.  I thought about schemes for
doing this, but none seemed as sensible as trying to build upon what IANA
already has.  The private & vendor-specific trees provide a mechanism by
which there's *at least* some record of a given format's origin, which is
clearly better than it being completely ad hoc.  The only downside I can
see is that two apps might start using the same media type, but with
different mappings to Matroska's syntactic structures (i.e. CodecPrivate
and CodecState).  Hopefully, any encoding common enough to have independent
implementations by multiple apps would've already been promoted to have an
official Matroska CodecID, whereupon such implementation details can be
specified.

> Although tempting I think it's better to avoid interleaving all kinds of
stuff with audio/video.

It's a valid and understandable position, but what attracted me to the
Matroska format was my understanding that it's an open and extensible
format.  If it's too resistant to innovation, then developers might seek
alternatives.  This is how file formats get superseded and fall into
disuse.  Therefore, I think it's important for Matroska to offer extension
mechanisms that strike the right balance between flexibility and
compatibility.

Matt

On Sun, Feb 26, 2017 at 11:29 AM, Steve Lhomme <[email-masked]> wrote:

> type