Subject: [Cellar] Matroska BlockDuration

From: Jerome Martinez <[email-masked]>
Date: Tue, 14 Nov 2017 19:11:09 +0100

Hello,

from http://ffmpeg.org/pipermail/ffmpeg-devel/2017-November/219642.html

there is a questioning (by myself) about the purpose of the following 
text in Matroska specs, about BlocckDuration:
"When set to 0 that means the frame is not a keyframe"

I don't understand this part. What is the relationship between being a 
key frame and the duration of the block?
I don't find an use case for setting 0 here, and what would be the 
expected result for a player.

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 16 Nov 2017 08:49:22 +0100

2017-11-14 19:11 GMT+01:00 Jerome Martinez <[email-masked]>:
> there is a questioning (by myself) about the purpose of the following text
> in Matroska specs, about BlocckDuration:
> I don't understand this part. What is the relationship between being a key
> frame and the duration of the block?
> I don't find an use case for setting 0 here, and what would be the expected
> result for a player.

I think this is a mistake. Indeed it doesn't make sense. So it should
be removed. Even a duration of 0 in itself doesn't seem to make much
sense. After all Blocks in Matroska don't really have a duration
unless you explicitly set it.

Which BTW should be something that should be clarified. Especially in
the context of preservation, ie you're supposed to display the frame
as long as there is another usable frame, unless there's a duration.
In which case what happens is undefined behaviour. Same thing for
audio. Should it revert to black/silence ?

> Jérôme
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 16 Nov 2017 09:01:42 +0100

On 16/11/2017 08:49, Steve Lhomme wrote:
> be removed.

OK. I'll send a patch.
Edit: in the meanwhile, I see the PR, you were quicker than me :).

> Even a duration of 0 in itself doesn't seem to make much

The test case was reported
https://ffmpeg.org/pipermail/ffmpeg-devel/2017-November/219945.html
It makes sense, as you can not know in advance how long is the subtitle 
(it is the case for PGS in the example, it would be same for e.g. 
608/708 closed captions real time transcoding to UTF-8 Plain Text), so 
in the case of streaming you can not wait up to know the duration of the 
block before writing/sending the subtitle block.
Is there something else in Matroska for saying "do not display anymore 
the previous content"?

> audio. Should it revert to black/silence ?

IMO it is the role of the player to decide what to show for "empty" 
stuff, and it is normal that the format does not define how to display 
no video (!= black, e.g. on a transparent screen it would be 100% 
transparent instead, and for any future technology you don't know what 
will be the way to display "no video") and no audio (!= silence).

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 16 Nov 2017 10:52:45 +0100

> https://ffmpeg.org/pipermail/ffmpeg-devel/2017-November/219945.html

Not sure what needs to be fixed in mkvalidator. A 0 BlockDuration
doesn't seem right. In fact I wonder if we should just set in the
specs that the field MUST NOT be 0. It has a default value (of
DefaultDuration) which cannot be 0 too.

Is there a case where it would make sense to set 0 ?

> It makes sense, as you can not know in advance how long is the subtitle (it
> is the case for PGS in the example, it would be same for e.g. 608/708 closed
> captions real time transcoding to UTF-8 Plain Text), so in the case of
> streaming you can not wait up to know the duration of the block before
> writing/sending the subtitle block.

If it's streaming then the issue already exists in the original
stream, you can't tell the duration until you receive the next one, if
I understand correctly. It seems you don't need to set the duration in
Matroska for such a case. The next "frame" will erase the previous
one.

> Is there something else in Matroska for saying "do not display anymore the
> previous content"?

Not for the moment. Does this exist anywhere else ? That sounds like
something that may be internal to the codec.

> IMO it is the role of the player to decide what to show for "empty" stuff,
> and it is normal that the format does not define how to display no video (!=
> black, e.g. on a transparent screen it would be 100% transparent instead,
> and for any future technology you don't know what will be the way to display
> "no video") and no audio (!= silence).
>
>
> Jérôme
>

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 16 Nov 2017 11:23:46 +0100

> Is there a case where it would make sense to set 0 ?

In the case there is nothing to show, e.g. a previous subtitle element 
which should disappear.

> Matroska for such a case. The next "frame" will erase the previous

The next frame is too far away. and has content.
Here, the issue is that the content is not linear, you have "holes" 
between frames.

One could provide an empty subtitle with no duration, but it is not 
exactly same as "no data" (e.g. black frame is a frame content, it is 
not the lack of frame).

IIUC, a duration of 0 is for saying "no more content starting now" in 
the use case. How to say that if mkvalidator says it is invalid?

An example:
1/ several frames with BlockDuration having the right value, content is 
correctly displayed.
2/ Last subtitle is displayed, it is file streaming and we don't know at 
this moment the duration, so BlockDuration is not filled (="value is 
assumed to be the difference between the timestamp of this Block and the 
timestamp of the next Block")
3/ At some point, we know that the subtitle stream should stop to be 
displayed (before the full duration indicated in the header). End of 
this stream. an empty subtitle with duration of 0 in order to say "this 
is the end of the stream", as the expected marker for the duration of 
the previous block.

With file on disk (no streaming), it is easy, BlockDuration in last 
subtitle block is used for setting the end of the subtitle track (step 2 
from above and there is no step 3), easy, but in the case of streaming 
there is no solution if I understand correctly, and a BlockDuration of 0 
could be a method for saying that this is the end of the previous block.

In other words, a BlockDuration of 0 is a method for saying "when there 
is a break in a track like for subtitle tracks" (in description of 
BlockDuration) when we do streaming without knowing in advance the 
duration of the previous block. I think it should not be marked as an 
error in mkvalidator.

I guess that a workaround is to have an empty subtitle block and to 
remove the BlockDuration when the goal is to clear the previous 
subtitle, just in the case of live streaming the meaning of the block is 
not really clear between "show black" and "this is a break in the track" 
as it is for differed time (with BlockDuration of the last frame). Not 
big impact, just some semantic in the case of live streaming.

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 17 Nov 2017 09:25:18 +0100

2017-11-16 11:23 GMT+01:00 Jerome Martinez <[email-masked]>:
> In the case there is nothing to show, e.g. a previous subtitle element which
> should disappear.

OK but "should disappear" it's too vague. How is it signaled in the
original content ? This signaling defines how it should be handled in
Matroska. If there's an actual element that says the result rendering
of the previous element should stop then we need it in Matroska. If
there's a duration, then it needs to be set.

>>> (it
> Here, the issue is that the content is not linear, you have "holes" between
> frames.

But if the original content has "holes" it needs to be preserved that
way. It may be intentional, it may be a bug in the source material but
how do you decide how it should be handled and where it should stop ?
You do not have that information in the first place.

> One could provide an empty subtitle with no duration, but it is not exactly
> same as "no data" (e.g. black frame is a frame content, it is not the lack
> of frame).
>
> IIUC, a duration of 0 is for saying "no more content starting now" in the
> use case. How to say that if mkvalidator says it is invalid?
>
> An example:
> this moment the duration, so BlockDuration is not filled (="value is assumed
> to be the difference between the timestamp of this Block and the timestamp
> of the next Block")
> displayed (before the full duration indicated in the header). End of this
> stream. an empty subtitle with duration of 0 in order to say "this is the
> end of the stream", as the expected marker for the duration of the previous
> block.

I don't think this is a good solution. On the muxer side you'd need to
know beforehand this is the last element of the stream. And as you
mentioned this may not even be an option when streaming.

Also if that element is damaged you'd end up with a stream that has
subtitles that never end ?

I agree we need to define how end of stream is handled for subtitles.
It's already done (maybe not in the specs) for video. The duration
should be set on the last video frame. The same thing should be done
for subtitles. But as seen above, that rule doesn't work if the last
element is damaged or the file truncated. So we should rather define
how to interpret data in case the duration is not known. Right now
it's a best effort approach. Most player will just use the audio as
basis and end playback with audio.

> With file on disk (no streaming), it is easy, BlockDuration in last subtitle
> block is used for setting the end of the subtitle track (step 2 from above
> and there is no step 3), easy, but in the case of streaming there is no
> solution if I understand correctly, and a BlockDuration of 0 could be a
> method for saying that this is the end of the previous block.
>
> In other words, a BlockDuration of 0 is a method for saying "when there is a
> break in a track like for subtitle tracks" (in description of BlockDuration)
> when we do streaming without knowing in advance the duration of the previous
> block. I think it should not be marked as an error in mkvalidator.
>
> I guess that a workaround is to have an empty subtitle block and to remove
> the BlockDuration when the goal is to clear the previous subtitle, just in
> the case of live streaming the meaning of the block is not really clear
> between "show black" and "this is a break in the track" as it is for
> differed time (with BlockDuration of the last frame). Not big impact, just
> some semantic in the case of live streaming.
>
>

---

From: Jerome Martinez <[email-masked]>
Date: Fri, 17 Nov 2017 09:42:36 +0100

On 17/11/2017 09:25, Steve Lhomme wrote:
> original content ?

For what I know (608/708), it is a "clear content" command if there is a 
still a stream (bytes are provided), or also the stream stops to be 
provided (i.e. 608 during content, no 608 bytes at all during ads, then 
608 again during content)
I understand that it could be considered as "this is a new frame with 
empty content so just send a black frame in your transcoding with no 
BlockDuration".
What is weird IMO is that you can indicate that the stream stops to be 
provided when you know it in advance (before the last frame so you can 
fill BlockDuration) but you can't indicate the exact same meaning when 
you don't know it in advance. IMO it is incoherency in the specs, with a 
"feature" available only in some cases (as Matroska is, if I understand 
well, aslo intended for streaming). "vague" or not is not the topic, as 
this issue is already there when you set BlockDuration.

> [...]
> for subtitles.

I don't think that there is any difference between video and subtitles 
(and audio). In all case it is easy if you know in advance (before 
writing the block) when it ends (you set BlockDuration), else it is not 
possible.
So I don't think that it is already done for video, it is the same issue 
(but no practical use case) for streaming.

***

Back to the practical issue: do you mean that in this case (hole in 
subtitles during file streaming), in order to have mkvalidator not 
complaining, the implementation should just put an empty subtitle with 
no BlockDuration, without doing any semantic difference between 
empty/black/silence content and hole in the stream?

Jérôme