Subject: [Cellar] EBML element identifier assignment

From: Tobias Rapp <[email-masked]>
Date: Thu, 7 Sep 2017 15:32:28 +0200

Hi,

just out of curiosity: How are EBML element identifiers chosen which are
used within Matroska? Often-used elements seem to get a shorter
identifier which in turn determines the most significant bits. Are the
remaining bits assigned randomly?

In case some vendor would like to add a new element (theoretical
example, I have no current plans), is there some "reserved range"
similar to the Private Use Area of Unicode?

Regards,
Tobias

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 24 Sep 2017 15:36:29 +0200

> remaining bits assigned randomly?

Elements with only 1 byte are usually only for elements that we are
sure will be used a lot in the format (outside of the default value).
Elements with 4 bytes are the higher level elements that we want to
seek to and recover quickly without false positive when there's damage
in the file.

The rest of the bits are no exactly random. When possible they use a
character or ASCII symbol that can be associated with what the element
is about. So when you look in binary it almost makes sense.

Some values that always go together also have incremental values from
one another.

> similar to the Private Use Area of Unicode?

No. It would be good to keep track of them in some ways through. Like
the DivX ones for example.

> Regards,
> Tobias
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Tobias Rapp <[email-masked]>
Date: Mon, 25 Sep 2017 16:28:02 +0200

> is about. So when you look in binary it almost makes sense.

Didn't notice the ASCII usage yet but will take a look the next time I 
make a MKV hex dump :-)

> the DivX ones for example.

I assume you refer to the list of EBML elements listed at 
http://labs.divx.com/node/16601 which seem to be part of the current 
EBML spec draft.

Is it useful to reserve some element ID range for "private use" and 
declare that it won't be used by future MKV versions?

Pros/Cons I see:
  (+) allows for more flexibility at vendors
  (-) might encourage fragmentation of the standard

Thus I'm not completely sure it's worth the work but just want to get 
the question out of my head and ask the list for opinions.

Regards,
Tobias

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 25 Sep 2017 17:07:09 +0200

2017-09-25 16:28 GMT+02:00 Tobias Rapp <[email-masked]>:
> Didn't notice the ASCII usage yet but will take a look the next time I make
> a MKV hex dump :-)
>
> http://labs.divx.com/node/16601 which seem to be part of the current EBML
> spec draft.
>
> Is it useful to reserve some element ID range for "private use" and declare
> that it won't be used by future MKV versions?
>
> Pros/Cons I see:

It might be a good idea to reserve some level 1, 2 and 4 elements for
future Matroska usage. This way we know they are not used by anyone.
It might be a little late for that though as we have no idea what's
out there in the wild.

> Thus I'm not completely sure it's worth the work but just want to get the
> question out of my head and ask the list for opinions.
>
> Regards,
> Tobias
>

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 25 Sep 2017 17:50:11 +0200

> It might be a little late for that though as we have no idea what's

Actually classic idea (from other specs) is to reserve some elements for 
future **not** Matroska usage a.k.a. explicitly flagged "private use".
All other elements would be forbidden if not in specs.
I suggest not to reserve any level 1 or 2 elements, as they are too rare 
for affording to lose them for private use.

Some ideas from other readings:
- we could assign 1 byte of a 4-byte element for private use i.e. 
[XX][00][00][00] to [XX][FF][FF][FF] are defined as for private use 
depending of the registry, and we decide about [XX] value in specs.
- we could assign a "registry" element (where? In EBML part after 
DocTypeReadVersion? Issue is if one wants to use private elements from 2 
registries), an entity willing to use private elements would have to 
write this element ant put its name in it (advising reverse DNS naming 
convention?), or we could say that the 2nd and 3rd bytes (2^16 
possibilities :) ) need to be registered to Matroska with name of the 
entity planning to use the private elements, before usage, or advising 
that they should be chosen trying to be unique (depending of their name?)

Main issue I see is that WebM is expanding elements when they want and 
we may have to try to find an agreement that such range is reserved if 
we want to be able to "backport" their addition to Matroska specs so 
WebM stays a subset of Matroska, but this is actually true for all other 
new elements too.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 25 Sep 2017 18:05:06 +0200

>> It might be a little late for that though as we have no idea what's

There's also the problem of 2 entities using the same reserved/free ID
for something different without knowing each other.

On the `Matroska Reader` side we should allow any ID we don't know
about. But maybe on the writing side the specs should be stricter on
what is allowed and what is not.

> I suggest not to reserve any level 1 or 2 elements, as they are too rare for
> affording to lose them for private use.

Level 1 are rare indeed. That doesn't mean a custom format should not
be allowed to use any. Even if there's only 1 or 2 allowed.

> [XX][00][00][00] to [XX][FF][FF][FF] are defined as for private use

Yes, maybe [XX]['P'][YY] and [XX]['P'][YY][YY] could be reserved
ranges. We have to find a range with a fancy code that doesn't collide
with any ID we have.

> registries), an entity willing to use private elements would have to write
> this element ant put its name in it (advising reverse DNS naming
> convention?), or we could say that the 2nd and 3rd bytes (2^16 possibilities
> :) ) need to be registered to Matroska with name of the entity planning to
> use the private elements, before usage, or advising that they should be
> chosen trying to be unique (depending of their name?)

The registry in the header is probably too much, or it should cover
the whole range of IDs (EBML v2 ?) or may a diff compared to a
specified version of the DocType.

But IMO it's better to have a public registry and tell people to
register their IDs. Similar to what exists for MP4 atoms. But I think
that would be something that goes through IETF/IANA.

> Main issue I see is that WebM is expanding elements when they want and we
> may have to try to find an agreement that such range is reserved if we want
> to be able to "backport" their addition to Matroska specs so WebM stays a
> subset of Matroska, but this is actually true for all other new elements
> too.

I'm sure if we setup some rules, they will follow them.

---

From: Tobias Rapp <[email-masked]>
Date: Tue, 26 Sep 2017 14:21:23 +0200

> for affording to lose them for private use.

I agree it should be this direction, not inverse.

> depending of the registry, and we decide about [XX] value in specs.

Sounds like a good idea.

> that they should be chosen trying to be unique (depending of their name?)

We should try to keep complexity low. Adding a recommendation like 
"element [XX][00][00][00] SHOULD contain a registry string in reverse 
DNS naming style when using elements from the private range" would be 
enough, in my view.

> WebM stays a subset of Matroska, but this is actually true for all other 

Don't know how such an agreement is typically handled in IETF 
specifications: as part of the spec itself, IANA, external, ...

Regards,
Tobias