Subject: Re: [Cellar] QuickTime timecode tracks in Matroska, was Ancillary data in Matroska

From: Steve Lhomme <[email-masked]>
Date: Wed, 26 Apr 2017 09:51:38 +0200

A bit late on the party but I agree with the move to a more specific
track type for timecodes rather than the generic ancillary data.

2017-03-20 22:07 GMT+01:00 Jerome Martinez <[email-masked]>:
> Le 20/03/2017 à 21:37, Dave Rice a écrit :
>>
>> [...]
>>>>
>>>> I’ve only seen a single value stored but then sometimes an edit list
>>>> used to alter the ordering of the timecode (in case when its
>>>> non-sequential). For Matroska possibly it could simply store a new timecode
>>>> Block at any case when it is not sequential.
>>>
>>> What about seeking?
>>
>> Isn't the seeking similar to QuickTime.
>
>
> Difference is that in QuickTime, a wrong "seek table" makes the playback
> impossible, so developers are more careful about it. and in the file I
> remember (I must still find it), there was only one offset in the offset
> table of QuickTime and a single "chunk" of time code with byte size of 4
> bytes per frame (to be confirmed as you say you saw edit lists instead)
>
>>   In QuickTime you'd need the offset table of the timecode track to
>> understand the number and location of all timecode samples. In Matroska you
>> would use the associated CuePoints of the timecode track for the same
>> reference. By parsing Cues the reader should be able to determine if the
>> timecode is stripped or not and if not then have the CueTimes for each
>> associated timecode Cluster.
>>
>>> If you have a timecode block at the first frame and also at the middle of
>>> the file, how a player can know that there is a timecode block breaking the
>>> sequential order without parsing the whole file when there is a seek request
>>> to the last minute of content? In that case, time code real value would be
>>> set to "waiting for new block" (and this block never comes). I don't think
>>> that forcing full parsing of the file with time code for seeking is a good
>>> solution.
>>
>> Same. But in the case of video if you seek to a P-frame then you have to
>> go backwards and decode from the I-frame. Similarly if seeking to a
>> timepoint without a timecode Cluster (analogous to a P-frame), then the
>> reader would have to use the Cues to decode the prior timecode Cluster
>> (analogous to I-frame) in order to determine the timecode value of the seek
>> point.
>
>
> Possible.
> But we need to be careful, what does it means?
> examples of issue:
> 1/ can a player consider that if there is only one CuePoint with CueTrack =
> the ID of the time code track, it means that this is a stripped content and
> time codes are in sequential order?
> 2/ can a player consider that if there are only 2 CuePoints with CueTrack =
> the ID of the time code track, it means that this is a stripped content and
> time codes are in sequential order except for one place and there is a
> single discontinuity?
> 3/ or must a player consider that if there only 2 CuePoints (let say frame 0
> and frame 1000) with CueTrack = the ID of the time code track, it means that
> it must seek to 0 if the requested frame is 999 (so a loooooong read of the
> file on disk before being able to play frame 999)
> If answers are 2/ yes 3/ no, I understand that the only method for storing a
> time code track with all values not sequential is to have a CuePoint for
> each time code frame (which makes the Cues huge, 15-20 bytes of CuePoint per
> time code frame).
>
> Please provide example about how you imagine CuePoints in the cases:
> 1/ sequential time codes during 2000 frames
> 2/ sequential time codes during 2000 frames except one discontinuity at
> frame 1000
> 3/ 2000 different times codes
> and where a player must seek if the request is to seek to frame 999.

Are timecode frames supposed to be matching a specific video frame or
audio frame ? Or they are loose ? If the former case it may be best to
be attached as a BlockAdditional. Otherwise they should always be in
sequential/display order, just like audio is. Even if video is not.

Having a separate track means the timestamps may not match exactly the
video or audio tracks. So in case of seeking/discontinuity it's like
you said, there's a state where you need to way for the next Timecode
block to know where you are. But in general a Timecode block
should/must be placed before the audio/video it corresponds to. It's
the same requirement there is in WebM that audio blocks are muxed
before video blocks. Timecode would be even more prioritary.

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 26 Apr 2017 10:25:21 +0200

It seems a lot of these formats include timecodes (or are basically
timecodes). There is still debate whether timecodes should have their
own track and ancillary data a different type of tracks. As expressed
in a different thread I think timecodes Blocks should come before the
video/audio block they are meant to match. Is it the same with
ancillary data ? Do ancillary data match a specific video frame
compared to timecodes that are more generally loose ?

I think this consideration will define how we should go forward
handling these data.

I agree with Jerome that we're not going to reinvent yet another
timecode format. We should support the main ones. Same thing for
ancillary data.

2017-02-24 5:18 GMT+01:00 Dave Rice <[email-masked]>:
> Hi all,
>
> On Jan 7, 2017, at 11:57 AM, Dave Rice <[email-masked]> wrote:
>
>
> […]
>
> On Dec 27, 2016, at 3:22 PM, Jerome Martinez <[email-masked]> wrote:
>
>
> […]
>
> "N_" prefix arbitrary used (2nd letter of "Ancillary" because first letter
> is already used)
>
>
> Please also add a patch for the TrackType Element in ebml_matroska.xml as
> we’ll need an unsigned integer to represent the ‘ancillary’ type of track.
>
> N_QUICKTIME: is a pure copy of V_QUICKTIME and A_QUICKTIME already supported
> in Matroska, N_ would be used which would be used for e.g. tmcd codec
> (QuickTime time code). Exactly same principles as video and audio. Note that
> classic usage is to store only the first frame content and other content is
> computed from it, so a player would have to read the first frame even if
> there is a direct seek request to another place; this could be avoided with
> some "hack" in the track header, could be the next step after this one is
> accepted.
>
>
> Would need a clear caveat in the definition that this implementation would
> only support storage of the first timecode value and computation of each
> additional value. A transwrap from a mov file with a timecode track that
> uses an edit list to handle non-continuous timecode could not be a lossless
> transwrap in this case, since the edit list would be lost and the new
> timecode track in the Matroska file would appear continuous.
>
> Also reference to tmcd atom is not clear enough. Is it tref/tmcd, gmhd/tmcd,
> stsd/tmcd.
>
> If this is stsd/tmcd (I presume), then what time scale should be used? The
> time scale stored in tmcd or the time scale of the Matroska track?
>
> I like that copying stsd/tmcd copies in the timecode label as well.
>
>
> I started a draft of a PR for defining QuickTime-style timecode in Matroska
> based on Jerome’s suggestion, see
> https://github.com/Matroska-Org/matroska-specification/pull/102. I left
> CodecPrivate as containing the entire Sample description table array of the
> stsd atom, rather than starting after the track type atom (such as the image
> descriptor, sound descriptor), because in the case of timecode, values of
> the tmcd are needed to interpret what the successive timecode values will
> be. The description of the Codec Mapping uses timecode as an example of
> ancillary data but not exclusively. Perhaps we should constrain what
> quicktime media types are relevant here, as several of the media types
> defined at
> https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-SW1
> could be considered as ancillary. Comments?
>
> Best Regards,
> Dave Rice
>
>

---

From: Dave Rice <[email-masked]>
Date: Fri, 28 Apr 2017 22:03:24 -0400

> On Apr 26, 2017, at 3:51 AM, Steve Lhomme <[email-masked]> wrote:
> 

I'd propose to limit the use case to the former. The idea of storing timecode data within BlockAdditional is interesting, but would that limit the metadata possibilities. For instance if timecode data is stored as it's own TrackEntry, then potentially a file may contain two timecode TrackEntries, one depicting timecode values from the original source materials used (likely non-continuous), whereas another TrackEntry could represent a new continuous timecode track that served some purpose in a video production. With these timecodes as two TrackEntries with their own Clusters, they could have distinct names and could be independently described with flags, so one may be enabled and default while the other is disabled.

With timecode in BlockAddition, there could potentially be two sets of timecode (two BlockMore), but they would be indistinguishable and could not be controlled or labelled independently.

> block to know where you are.

This is the case if the there's one timecode value stored per video frame, but this may not be the case. We also discussed using QuickTime's strategy of storing an integer as a timecode while the timecode track's private data notes what equation to use to convert the integer into a timecode (flags like max24, timebase, etc). In this case a timecode value would only need to be stored at a point in time when the timecode is non-continuous.

> But in general a Timecode block

Does Matroska have the same ordering requirement? I don't see them. Is it valid to store Clusters in reverse timestamp order?

Dave Rice