Subject: Re: [Cellar] QuickTime timecode tracks in Matroska, was Ancillary data in Matroska

From: Dave Rice <[email-masked]>
Date: Tue, 14 Mar 2017 15:14:31 -0400

> On Feb 23, 2017, at 11:18 PM, Dave Rice <[email-masked]> wrote:
> 
> Hi all,
> 
>> On Jan 7, 2017, at 11:57 AM, Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
> 
> […]
> 
>>> On Dec 27, 2016, at 3:22 PM, Jerome Martinez <[email-masked] <mailto:[email-masked]>> wrote:
> 
> […]
> 
>>> "N_" prefix arbitrary used (2nd letter of "Ancillary" because first letter is already used)
>> 
>> Please also add a patch for the TrackType Element in ebml_matroska.xml as we’ll need an unsigned integer to represent the ‘ancillary’ type of track.
>> 
>>> N_QUICKTIME: is a pure copy of V_QUICKTIME and A_QUICKTIME already supported in Matroska, N_ would be used which would be used for e.g. tmcd codec (QuickTime time code). Exactly same principles as video and audio. Note that classic usage is to store only the first frame content and other content is computed from it, so a player would have to read the first frame even if there is a direct seek request to another place; this could be avoided with some "hack" in the track header, could be the next step after this one is accepted.
>> 
>> Would need a clear caveat in the definition that this implementation would only support storage of the first timecode value and computation of each additional value. A transwrap from a mov file with a timecode track that uses an edit list to handle non-continuous timecode could not be a lossless transwrap in this case, since the edit list would be lost and the new timecode track in the Matroska file would appear continuous.
>> 
>> Also reference to tmcd atom is not clear enough. Is it tref/tmcd, gmhd/tmcd, stsd/tmcd.
>> 
>> If this is stsd/tmcd (I presume), then what time scale should be used? The time scale stored in tmcd or the time scale of the Matroska track?
>> 
>> I like that copying stsd/tmcd copies in the timecode label as well.
> 
> I started a draft of a PR for defining QuickTime-style timecode in Matroska based on Jerome’s suggestion, see https://github.com/Matroska-Org/matroska-specification/pull/102 <https://github.com/Matroska-Org/matroska-specification/pull/102>. I left CodecPrivate as containing the entire Sample description table array of the stsd atom, rather than starting after the track type atom (such as the image descriptor, sound descriptor), because in the case of timecode, values of the tmcd are needed to interpret what the successive timecode values will be. The description of the Codec Mapping uses timecode as an example of ancillary data but not exclusively. Perhaps we should constrain what quicktime media types are relevant here, as several of the media types defined at https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-SW1 <https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-SW1> could be considered as ancillary. Comments?

After looking at this some more, I decided to create an alternate proposal for a T_QUICKTIME codec mapping, rather than to store QuickTime-style timecode in the more vague concept of an ancillary N_QUICKTIME. See the pull request at https://github.com/Matroska-Org/matroska-specification/pull/112 <https://github.com/Matroska-Org/matroska-specification/pull/112>.

For reference the proposed definition of T_QUICKTIME is:
{
Codec ID: T_QUICKTIME

Codec Name: QuickTime timecode track

Description: `T_QUICKTIME` is used to identify Timecode tracks as stored in QuickTime. The `QuickTime Timecode Sample Data` is stored within the Block's data section. For an explanation of `QuickTime Timecode Sample Data` read [QuickTime File Format Specification](https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-57421).

Initialisation: The `Private Data` contains the `Timecode Sample Description` which in QuickTime is stored within the 'tmcd' atom after the mandatory `data reference index` value. For an explanation of the `Timecode Sample Description` read [QuickTime File Format Specification](https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html#//apple_ref/doc/uid/TP40000939-CH205-69831).
}

Attached is a draft Matroska file with an T_QUICKTIME style timecode track.

The CodecPrivate section uses the QuickTime Timecode Sample Data which for this attachment is 000000000000000100007530000003E91E00000000136E616D650007000054415045233534, which equates to:

00000000        0           Resserved
00000001        1           Timecode flags
00007530        30000       Time scale
000003E9        1001        Frame duration
1E              30          Number of frames
00              0           Reserved
00000013        19          Size of name atom
6E616D65        name        Fourcc of name atom
 0007           7           Size of Timecode Label
 0000           en          Language of Timecode Label
 54415045233534 TAPE#54     Timecode Label

This example includes a timecode label, the shortest possible Private Data for T_QUICKTIME would be 18 bytes:

00000000        0           Resserved
00000001        1           Timecode flags
00007530        30000       Time scale
000003E9        1001        Frame duration
1E              30          Number of frames
00              0           Reserved

The attachment contains a SimpleBlock using the T_QUICKTIME with a example value of 0x0001fc8c, which according to the QuickTime Timecode Sample Data is 01:12:23;28 and subsequent timecode values for the track would be calculated according to the incrementation defined in the Private Data.

Comments?
Dave Rice

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 14 Mar 2017 20:39:56 +0100

Le 14/03/2017 à 20:14, Dave Rice a écrit :
> [...]
>
> After looking at this some more, I decided to create an alternate 
> proposal for a T_QUICKTIME codec mapping, rather than to store 
> QuickTime-style timecode in the more vague concept of an ancillary 
> N_QUICKTIME.

Trying to find some counter-arguments, I have to agree that this may be 
better as we don't have any real example for usage of a generic 
QuickTime ancillary track.

>
> [...]
>
> The attachment contains a SimpleBlock using the T_QUICKTIME with a 
> example value of 0x0001fc8c, which according to the QuickTime Timecode 
> Sample Data is 01:12:23;28 and subsequent timecode values for the 
> track would be calculated according to the incrementation defined in 
> the Private Data.

Content location is another issue.
Time code may be stripped (only the first time code is stored and other 
time code must be computed from it + header metadata), and there is no 
way to know it in Matroska : QuickTime has the list of blocks in the 
track header so the player knows if the the time code is stripped (1 
block of 4 bytes) or not (more than 1 block), this is not the  case with 
Matroska.
How could we indicate to the player that the time code is stripped 
(=only one value stored, then player should compute value of other time 
code frames)?

Jérôme

---

From: Dave Rice <[email-masked]>
Date: Tue, 14 Mar 2017 15:45:41 -0400

> On Mar 14, 2017, at 3:39 PM, Jerome Martinez <[email-masked]> wrote:
> 
>> After looking at this some more, I decided to create an alternate proposal for a T_QUICKTIME codec mapping, rather than to store QuickTime-style timecode in the more vague concept of an ancillary N_QUICKTIME.
> 
> Trying to find some counter-arguments, I have to agree that this may be better as we don't have any real example for usage of a generic QuickTime ancillary track.
> 
>> 
>> [...]
>> 
> Time code may be stripped (only the first time code is stored and other time code must be computed from it + header metadata), and there is no way to know it in Matroska : QuickTime has the list of blocks in the track header so the player knows if the the time code is stripped (1 block of 4 bytes) or not (more than 1 block), this is not the  case with Matroska.
> How could we indicate to the player that the time code is stripped (=only one value stored, then player should compute value of other time code frames)?

Since the update proposal defines a new media type (timecode) it is possible to define a Timecode Master element (as similar done with Audio and Video) to store needed context.

OTOH I don’t think I’ve ever seen a QuickTime timecode track that was non-stripped. I’ve only seen a single value stored but then sometimes an edit list used to alter the ordering of the timecode (in case when its non-sequential). For Matroska possibly it could simply store a new timecode Block at any case when it is not sequential.
Dave

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 14 Mar 2017 20:57:19 +0100

Le 14/03/2017 à 20:45, Dave Rice a écrit :
> [...]
>
>> Time code may be stripped (only the first time code is stored and 
>> other time code must be computed from it + header metadata), and 
>> there is no way to know it in Matroska : QuickTime has the list of 
>> blocks in the track header so the player knows if the the time code 
>> is stripped (1 block of 4 bytes) or not (more than 1 block), this is 
>> not the case with Matroska.
>> (=only one value stored, then player should compute value of other 
>> time code frames)?
>
>
> Since the update proposal defines a new media type (timecode) it is 
> possible to define a Timecode Master element (as similar done with 
> Audio and Video) to store needed context.
>
> OTOH I don’t think I’ve ever seen a QuickTime timecode track that was 
> non-stripped.

OK, I need to check my files but I am pretty sure I have some.

> I’ve only seen a single value stored but then sometimes an edit list 
> used to alter the ordering of the timecode (in case when its 
> non-sequential). For Matroska possibly it could simply store a new 
> timecode Block at any case when it is not sequential.

What about seeking?
If you have a timecode block at the first frame and also at the middle 
of the file, how a player can know that there is a timecode block 
breaking the sequential order without parsing the whole file when there 
is a seek request to the last minute of content? In that case, time code 
real value would be set to "waiting for new block" (and this block never 
comes). I don't think that forcing full parsing of the file with time 
code for seeking is a good solution.

---

From: Kieran O Leary <[email-masked]>
Date: Tue, 14 Mar 2017 19:59:31 +0000

Hi

On Tue, Mar 14, 2017 at 7:45 PM, Dave Rice <[email-masked]> wrote:

> non-stripped. I’ve only seen a single value stored but then sometimes an
> edit list used to alter the ordering of the timecode (in case when its
> non-sequential). For Matroska possibly it could simply store a new timecode
> Block at any case when it is not sequential.
> Dave
>
>
Same here actually, I've tried Control Room/Media Express/FCP and it's
always striped. I'm sure there's some hidden option or different way of
connecting the deck to the capture card to achieve this, but timecode jumps
are always ignored on any capture I've seen, and the increment is applied
to the starting timecode.
Also i'd love to see your samples, Jerome.

-Kieran.

>
> _______________________________________________
>
>

---

From: Dave Rice <[email-masked]>
Date: Mon, 20 Mar 2017 16:37:02 -0400

> On Mar 14, 2017, at 3:57 PM, Jerome Martinez <[email-masked]> wrote:
> 
>> OTOH I don’t think I’ve ever seen a QuickTime timecode track that was non-stripped.
> 
>> I’ve only seen a single value stored but then sometimes an edit list used to alter the ordering of the timecode (in case when its non-sequential). For Matroska possibly it could simply store a new timecode Block at any case when it is not sequential.
> 
> What about seeking?

Isn't the seeking similar to QuickTime. In QuickTime you'd need the offset table of the timecode track to understand the number and location of all timecode samples. In Matroska you would use the associated CuePoints of the timecode track for the same reference. By parsing Cues the reader should be able to determine if the timecode is stripped or not and if not then have the CueTimes for each associated timecode Cluster.

> If you have a timecode block at the first frame and also at the middle of the file, how a player can know that there is a timecode block breaking the sequential order without parsing the whole file when there is a seek request to the last minute of content? In that case, time code real value would be set to "waiting for new block" (and this block never comes). I don't think that forcing full parsing of the file with time code for seeking is a good solution.

Same. But in the case of video if you seek to a P-frame then you have to go backwards and decode from the I-frame. Similarly if seeking to a timepoint without a timecode Cluster (analogous to a P-frame), then the reader would have to use the Cues to decode the prior timecode Cluster (analogous to I-frame) in order to determine the timecode value of the seek point.

Dave Rice

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 20 Mar 2017 22:07:29 +0100

Le 20/03/2017 à 21:37, Dave Rice a écrit :
> Isn't the seeking similar to QuickTime.

Difference is that in QuickTime, a wrong "seek table" makes the playback 
impossible, so developers are more careful about it. and in the file I 
remember (I must still find it), there was only one offset in the offset 
table of QuickTime and a single "chunk" of time code with byte size of 4 
bytes per frame (to be confirmed as you say you saw edit lists instead)

>   In QuickTime you'd need the offset table of the timecode track to understand the number and location of all timecode samples. In Matroska you would use the associated CuePoints of the timecode track for the same reference. By parsing Cues the reader should be able to determine if the timecode is stripped or not and if not then have the CueTimes for each associated timecode Cluster.
>

Possible.
But we need to be careful, what does it means?
examples of issue:
1/ can a player consider that if there is only one CuePoint with 
CueTrack = the ID of the time code track, it means that this is a 
stripped content and time codes are in sequential order?
2/ can a player consider that if there are only 2 CuePoints with 
CueTrack = the ID of the time code track, it means that this is a 
stripped content and time codes are in sequential order except for one 
place and there is a single discontinuity?
3/ or must a player consider that if there only 2 CuePoints (let say 
frame 0 and frame 1000) with CueTrack = the ID of the time code track, 
it means that it must seek to 0 if the requested frame is 999 (so a 
loooooong read of the file on disk before being able to play frame 999)
If answers are 2/ yes 3/ no, I understand that the only method for 
storing a time code track with all values not sequential is to have a 
CuePoint for each time code frame (which makes the Cues huge, 15-20 
bytes of CuePoint per time code frame).

Please provide example about how you imagine CuePoints in the cases:
1/ sequential time codes during 2000 frames
2/ sequential time codes during 2000 frames except one discontinuity at 
frame 1000
3/ 2000 different times codes
and where a player must seek if the request is to seek to frame 999.