Subject: Re: [Cellar] CodecIDs for non-standard and unregistered data formats

From: Steve Lhomme <[email-masked]>
Date: Wed, 26 Apr 2017 10:58:47 +0200

2017-02-26 17:59 GMT+01:00 Jerome Martinez <[email-masked]>:
> Le 26/02/2017 à 17:29, Steve Lhomme a écrit :
>>
>> 2017-02-14 7:46 GMT+01:00 Matt Gruenke <[email-masked]>:
>>>
>>> Related to my recent query about TrackTypes for non-standard data
>>> formats,
>>> I'd like to discuss CodecIDs.  I had previously raised this issue, on the
>>> matroska-devel list, and was advised to take it up, here.  In the
>>> interim,
>>> my thinking on the matter has evolved.
>>>
>>> I wonder why not allow IANA Media Types (aka MIME types), as a sub-class
>>> of
>>> CodecIDs?  Of course, the official Matroska CodecID would be preferred,
>>> for
>>> all formats for which one has been designated.  However, this would
>>> enable
>>> broader use of Matroska files, including for private,
>>> application-specific
>>> data stream formats.
>>
>> The main goal of Matroska is multimedia interleaved/timestamped data.
>> It's possible to have "complex" track and put in it whatever you want.
>> If a timestamp is required. If it's not, using attachment is what you
>> want and it already has a MIME type.
>>
>>> Perhaps a prefix like 'X_MEDIATYPE/' could simply be prepended to media
>>> type
>>> names conforming with RFC 6838.  The value of TrackType would still
>>> determine whether video, audio, or any other type-specific elements can
>>> be
>>> used in such tracks.
>>
>> Although tempting I think it's better to avoid interleaving all kinds
>> of stuff with audio/video. You could want to embed a "video/quicktime"
>> but it would not be interleaved, or you'd need to parse/mux the data
>> and then it's not a clean "video/quicktime" you get anymore.
>
>
> I don't think this is the point.
> Let's imagine we don't have any AVI compatibility layer (for whatever
> reason, e.g. lawyers convinced a country that BITMAPINFOHEADER /
> WAVEFORMATEX structure are patented), we have no other way for having e.g.
> AMR, G719, G729 and so on because it is not defined in native Matroska codec
> identifiers, but IANA media types have them (so a muxer could use
> "X_MEDIATYPE/audio/G729").
> Consider Matt suggestion as similar to the AVI compatibility layer: this is
> just another way to define a codec identifier, also for
> interleaved/timestamped video and audio content.
>
> Right now, I am thinking to TTML fragments (subtitles), instead of defining
> any TTML stuff in Matroska specs, "X_MEDIATYPE/application/ttml+xml" could
> be used and we rely on
> https://www.iana.org/assignments/media-types/application/ttml+xml .
> For crazy ideas but we should not forbid crazy ideas :), someone may want to
> use PDF as a video format with one PDF page per video frame (similar to
> MJPEG), and instead of defining and requesting a "V_PDF" codec identifier,
> one would use "X_MEDIATYPE/application/pdf".

But if the goal is to be played as a video, it should use the video
track type. And so it should probably use a V_SOMETHING name.

I don't think there's anything in the specs that forbid using your own
V_SOMETHING name. Adding V_MIME/your-mime/here may be a good idea
though.

> At long term, a potential decision would be to delegate codec identifiers
> management to IANA as they already do the job (for RTP and so on) in order
> to not duplicate efforts, and use Matroska list only when IANA does not fit
> Matroska needs, i.e. why should we care of having a spec about
> "V_MPEG4/ISO/AVC" (note: it is not on the codec spec page but a lot used) if
> we could rely on "video/H264" registration? (this is theoretical,
> "V_MPEG4/ISO/AVC" is already used. but we could do that for upcoming AV1
> defined by IETF).
>
> Jérôme

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 26 Apr 2017 11:13:33 +0200

2017-02-27 4:04 GMT+01:00 Matt Gruenke <[email-masked]>:
> Let's take this example of a camera having both a visible light sensor and a
> depth sensor, as well as a stereo microphone.  For various reasons, its
> depth sensor captures frames at different times and a different rate than
> its color image sensor.  I'd like to record a video file containing an audio
> track, a normal video track, and a depth track.  I'd like the audio and
> video portions to be playable in normal video player apps.  When using
> enhanced apps, the depth data may also be utilized.

It seems that the depth data cannot be used on their own and would
need to be coupled with the video frames. So that would probably go as
a BlockAddition data (or something similar as it's currently tied to
the Track codec). Even if they are repeated for each video frame which
the sensor doesn't change the value.

> Today, most players I've tried will behave as I've described, if I simply
> write the depth data using an unspecified TrackType and having a custom
> CodecID.  However, not all players handle it gracefully (or at all), and
> there's no guarantee that the others will continue to work.
>
> Having a dedicated TrackType (or range, thereof) would solve one potential
> problem.  Perhaps the Ancillary TrackType already meets this need.
>
> Using nonstandard CodecIDs is another concern.  I thought about schemes for
> doing this, but none seemed as sensible as trying to build upon what IANA
> already has.  The private & vendor-specific trees provide a mechanism by
> which there's at least some record of a given format's origin, which is
> clearly better than it being completely ad hoc.  The only downside I can see
> is that two apps might start using the same media type, but with different
> mappings to Matroska's syntactic structures (i.e. CodecPrivate and
> CodecState).  Hopefully, any encoding common enough to have independent
> implementations by multiple apps would've already been promoted to have an
> official Matroska CodecID, whereupon such implementation details can be
> specified.

It seems to me it should not be a different track, so I'll use this
hypothesis. We may need to add more elements to the Track to define
what each BlockAdditionID is, maybe using a MIME type or something
similar. Maybe a type like depth data or extra lossless complement
(for the only case I know of using BlockAddition: lossy part in the
regular Block and lossless complement in the BlockAddition).

>> Although tempting I think it's better to avoid interleaving all kinds of
>> stuff with audio/video.
>
> It's a valid and understandable position, but what attracted me to the
> Matroska format was my understanding that it's an open and extensible
> format.  If it's too resistant to innovation, then developers might seek
> alternatives.  This is how file formats get superseded and fall into disuse.
> Therefore, I think it's important for Matroska to offer extension mechanisms
> that strike the right balance between flexibility and compatibility.

You're right, it's extensible. But it would be wrong if people extend
it in a bogus way. For example non-interleaved data shouldn't go in
the interleaved parts of the format. There are already all kinds of
lesser known mechanism to extend Matroska in a clean way. We must
emphasis on that.

>
> Matt
>
>
> On Sun, Feb 26, 2017 at 11:29 AM, Steve Lhomme <[email-masked]> wrote:
>>
>> > Thank you for considering my suggestion.
>> >
>> >
>> > Matt
>> >
>> >
>> > _______________________________________________
>> > Cellar mailing list
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/cellar
>> >
>>
>>
>>
>> --
>> Steve Lhomme

---

From: Matt Gruenke <[email-masked]>
Date: Thu, 27 Apr 2017 23:16:41 -0400

> the sensor doesn't change the value.

I'd say it's no more or less coupled than audio.  If a movie has an audio
track, people will typically watch the video with the audio, but sometimes
without.  They rarely listen to the audio without the video, but they could
(and some do).  I think it's a good analogy for depth - it's an optional
augmentation of the video and could be paired with one of several different
video tracks (e.g. if the same file has a depth track and video from both a
visible light & a thermal sensor).

More importantly, if the depth frames are taken at slightly different times
than video, it's critical they have timestamps which reflect this.
Furthermore, it would be a really problematic to repeat the same data on
subsequent frames, in cases where no new depth frames have been captured in
between.  These kinds of sins some people committed with video created
substantial headaches for video processing.

Depth must be synchronized with video as precisely as possible.  Any
fudging of timestamps or duplication of data will interfere with attempts
to interpolate the depth frames to match the video (or vice versa).  It
might not be feasible or desirable to perform this interpolation at capture
time.

Matt