From nobody Mon Nov 9 10:19:16 2015 Return-Path: X-Original-To: cellar@ietfa.amsl.com Delivered-To: cellar@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 45E401A9113 for ; Mon, 9 Nov 2015 10:19:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 1.58 X-Spam-Level: * X-Spam-Status: No, score=1.58 tagged_above=-999 required=5 tests=[BAYES_50=0.8, HTML_MESSAGE=0.001, SPF_NEUTRAL=0.779] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xUPEDelbIpEH for ; Mon, 9 Nov 2015 10:19:11 -0800 (PST) Received: from s172.web-hosting.com (s172.web-hosting.com [68.65.122.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 293741A910A for ; Mon, 9 Nov 2015 10:19:10 -0800 (PST) Received: from [146.96.19.240] (port=23903 helo=[10.10.202.53]) by server172.web-hosting.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.85) (envelope-from ) id 1Zvr24-0012kr-OO; Mon, 09 Nov 2015 13:19:10 -0500 Content-Type: multipart/alternative; boundary="Apple-Mail=_EC8505EF-8992-48D3-AF40-A232BCBB9B0F" Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\)) From: Dave Rice In-Reply-To: Date: Mon, 9 Nov 2015 13:19:07 -0500 Message-Id: <5606B89B-FCF0-4C75-BAB8-FB1E212F8D82@dericed.com> References: <21E28D45-E45F-4CBE-AC3D-6E41DCE172B9@dericed.com> <20150828065002.GH3813@bunkus.org> To: Discussion about the current and future development of Matroska X-Mailer: Apple Mail (2.1990.1) X-OutGoing-Spam-Status: No, score=-1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server172.web-hosting.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - dericed.com X-Get-Message-Sender-Via: server172.web-hosting.com: authenticated_id: dave@dericed.com X-Source: X-Source-Args: X-Source-Dir: X-From-Rewrite: unmodified, already matched Archived-At: Cc: cellar@ietf.org Subject: Re: [Cellar] [Matroska-devel] EBML Schema X-BeenThere: cellar@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Codec Encoding for LossLess Archiving and Realtime transmission List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Nov 2015 18:19:14 -0000 --Apple-Mail=_EC8505EF-8992-48D3-AF40-A232BCBB9B0F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi all, > On Oct 3, 2015, at 9:46 AM, Steve Lhomme wrote: > On Aug 28, 2015 17:00, "Dave Rice" > wrote: > > > > Hi, > > > >> On Aug 28, 2015, at 2:50 AM, Moritz Bunkus > wrote: > >> > >> Hey, > >> > >> I have no objections, however I don't know a lot about XML schemas = in > >> the first place (neither about DTDs, to be honest). > > > > > > Honestly, I know a lot more about XML Schemas than I do about DTDs. = As wikipedia mentions at = https://en.wikipedia.org/wiki/Document_type_definition = , DTDs have = largely been superseded by XML Schemas. And at this point I think that = XML Schemas may be a more familiar analogy to use. > > > > I think XML Schemas also share more in common with specdata.xml than = DTDs do. Schemas use the node and have maxOccurs and minOccurs = attributes (specdata has semantically the same thing with mandatory and = multiple), they both have a similar declaration of element type, element = name and element description. Actually I think a semantically equivalent = version of specdata.xml could be written as an XML Schema. > > > > XML Schemas also offer a few advantages for machine readable = expressions; for instance XML Schemas can mandate a particular pattern = or regex for a value.=20 > > > >>> I propose the specdata.xml file here > >>> = https://github.com/Matroska-Org/foundation-source/blob/master/spectool/spe= cdata.xml = > >>> = > > >>> is a good basis for the consideration of an EBML Schema. =46rom = what I > >>> can see, specdata.xml is an expression of the EBML + Matroska > >>> specifications to support automated creation of documentation, but = the > >>> structure of this already shares a lot of similarity to XML = Schemas. > >> > >> > >> For both documentation (e.g. the table on the matroska.org = specs page is > >> generated from this file) and code (libMatroska's class hierarchy = is > >> generated automatically from this file) actually. > > > > > > Does specdata.xml play a role in mkvalidate? I'm thinking of the = potential to have an ebmlvalidator where you can provide the EBML Schema = to validate particular EBML docType. >=20 > Well the parsing code is generated from the XML file, so in a way, = yes. But it's not parsed "live". >=20 > >>> Is there a preference in handling the standardization of Matroska: > >>> documenting it in a similar fashion to our work in the EBML spec = or to > >>> define what an EBML Schema is and consider matroska an expression = of > >>> it? > >> > >> > >> I'm not sure whether or not I understand the implications. But my = gut > >> feeling is that having a definition for an EBML Schema would = benefit > >> other formats than Matroska, too, therefore the latter seems the = way to > >> go. > > > > > > I have the same feeling: > > - document EBML as a specification that includes rules for defining = a docType in the form of an EBML Schema > > - write an EBML Schema (updated specdata.xml) for Matroska and maybe = webM > > > >>> Are some changes to specdata.xml acceptable? Such as a filename = change > >>> or changing the name of the element of some attributes? > >> > >> > >> Well, like I said above the specdata.xml is used for generating = both > >> documentation and code. Both should stay viable. If changes to it = are > >> made then the accompanying tools must be updated as well. > >> > >>> Neither the current EBML specs nor the specdata.xml specifically = refer > >>> to the hierarchical arrangement of the elements, but this could be > >>> presumed by their ordering. For instance, could any level 3 = element be > >>> a child of any level 2 Master-element? I presume not, but I don't > >>> think it's clear anywhere what parent-child relationships are > >>> feasible. Possibly specdata.xml and/or the EBML Schema Definition > >>> could define the relationship between levels of related elements > >>> similar to how an XML Schema (XSD) does. > >> > >> > >> So far it is understood that an element not marked as a global = element > >> must only occur as a child of its parent. Its parent is the last = element > >> located before the child element in the specdata file with a lower = level > >> than the child element. Or something like that. > > > > > > This will need some documentation. That's how I've understood the = mkv spec as well but the definition for how an EBML Schema works should = be explicit about this. >=20 Any more opinion about how to go about (or if to go about) modifying = specdata.xml towards becoming an expression of a to-be-defined EBML = Schema for matroska and webm? As a summary of proposed changes to = specdata.xml - change to XML Schema conventions where relevant: - use maxOccurs attribute instead of the current = Multiple attribute. - use minOccurs attribute instead of the current = Mandatory attribute. - move documentation of elements to a sub-element = (allows for possible internationalization in the schema and better = semantics) - arrange elements in hierarchical form to indicate parent-child = relationships (rather than the current practices where all elements are = defined at the same level, and you have to parse back in elements to the = one with the lower-numbered level attribute to find the parent) A draft of specdata.xml with these changes is at = https://gist.github.com/dericed/f0a4bb0e7dc635ed1347 = . I can continue = to work on this and send back changes for advice/approval but if I do so = is there someone who could later update the tools that use specdata.xml = so that newly-defined EBML Schemas may later to be into use? btw I'm cc'ing the newly-established CELLAR listserv which focuses on = work on the EBML and Matroska specification. If you are interested in = these topics please considering subscribing at = https://www.ietf.org/mailman/listinfo/cellar = . Best Regards, Dave Rice --Apple-Mail=_EC8505EF-8992-48D3-AF40-A232BCBB9B0F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Hi all,

On Oct 3, 2015, at 9:46 AM, = Steve Lhomme <slhomme@matroska.org> wrote:

On Aug 28, 2015 17:00, "Dave Rice" <dave@dericed.com> wrote:
>
> Hi,
>
>> On Aug 28, 2015, at 2:50 AM, Moritz Bunkus <moritz@bunkus.org> = wrote:
>>
>> Hey,
>>
>> I have no objections, however I don't know a lot about XML = schemas in
>> the first place (neither about DTDs, to be honest).
>
>
> Honestly, I know a lot more about XML Schemas than I do about DTDs. = As wikipedia mentions at https://en.wikipedia.org/wiki/Document_type_definition, = DTDs have largely been superseded by XML Schemas. And at this point I = think that XML Schemas may be a more familiar analogy to use.
>
> I think XML Schemas also share more in common with specdata.xml = than DTDs do. Schemas use the <element> node and have maxOccurs = and minOccurs attributes (specdata has semantically the same thing with = mandatory and multiple), they both have a similar declaration of element = type, element name and element description. Actually I think a = semantically equivalent version of specdata.xml could be written as an = XML Schema.
>
> XML Schemas also offer a few advantages for machine readable = expressions; for instance XML Schemas can mandate a particular pattern = or regex for a value. 
>
>>> I propose the specdata.xml file here
>>> https://github.com/Matroska-Org/foundation-source/blob/master/s= pectool/specdata.xml
>>> <https://github.com/Matroska-Org/foundation-source/blob/master/s= pectool/specdata.xml>
>>> is a good basis for the consideration of an EBML Schema. = =46rom what I
>>> can see, specdata.xml is an expression of the EBML + = Matroska
>>> specifications to support automated creation of = documentation, but the
>>> structure of this already shares a lot of similarity to XML = Schemas.
>>
>>
>> For both documentation (e.g. the table on the matroska.org specs page = is
>> generated from this file) and code (libMatroska's class = hierarchy is
>> generated automatically from this file) actually.
>
>
> Does specdata.xml play a role in mkvalidate? I'm thinking of the = potential to have an ebmlvalidator where you can provide the EBML Schema = to validate particular EBML docType.

Well = the parsing code is generated from the XML file, so in a way, yes. But = it's not parsed "live".

>>> Is = there a preference in handling the standardization of Matroska:
>>> documenting it in a similar fashion to our work in the EBML = spec or to
>>> define what an EBML Schema is and consider matroska an = expression of
>>> it?
>>
>>
>> I'm not sure whether or not I understand the implications. But = my gut
>> feeling is that having a definition for an EBML Schema would = benefit
>> other formats than Matroska, too, therefore the latter seems = the way to
>> go.
>
>
> I have the same feeling:
> - document EBML as a specification that includes rules for defining = a docType in the form of an EBML Schema
> - write an EBML Schema (updated specdata.xml) for Matroska and = maybe webM
>
>>> Are some changes to specdata.xml acceptable? Such as a = filename change
>>> or changing the name of the <table> element of some = attributes?
>>
>>
>> Well, like I said above the specdata.xml is used for generating = both
>> documentation and code. Both should stay viable. If changes to = it are
>> made then the accompanying tools must be updated as well.
>>
>>> Neither the current EBML specs nor the specdata.xml = specifically refer
>>> to the hierarchical arrangement of the elements, but this = could be
>>> presumed by their ordering. For instance, could any level 3 = element be
>>> a child of any level 2 Master-element? I presume not, but I = don't
>>> think it's clear anywhere what parent-child relationships = are
>>> feasible. Possibly specdata.xml and/or the EBML Schema = Definition
>>> could define the relationship between levels of related = elements
>>> similar to how an XML Schema (XSD) does.
>>
>>
>> So far it is understood that an element not marked as a global = element
>> must only occur as a child of its parent. Its parent is the = last element
>> located before the child element in the specdata file with a = lower level
>> than the child element. Or something like that.
>
>
> This will need some documentation. That's how I've understood the = mkv spec as well but the definition for how an EBML Schema works should = be explicit about this.

Any more = opinion about how to go about (or if to go about) modifying specdata.xml = towards becoming an expression of a to-be-defined EBML Schema for = matroska and webm? As a summary of proposed changes to = specdata.xml

- change to XML Schema = conventions where relevant:
- use maxOccurs attribute = instead of the current Multiple attribute.
- = use minOccurs attribute instead of the current Mandatory = attribute.
- move documentation of = elements to a sub-element (allows for possible internationalization in = the schema and better semantics)
- arrange elements in = hierarchical form to indicate parent-child relationships (rather than = the current practices where all elements are defined at the same level, = and you have to parse back in elements to the one with the = lower-numbered level attribute to find the parent)

A draft of specdata.xml with these changes is = at https://gist.github.com/dericed/f0a4bb0e7dc635ed1347. I = can continue to work on this and send back changes for advice/approval = but if I do so is there someone who could later update the tools that = use specdata.xml so that newly-defined EBML Schemas may later to be into = use?

btw I'm cc'ing the = newly-established CELLAR listserv which focuses on work on the EBML and = Matroska specification. If you are interested in these topics please = considering subscribing at https://www.ietf.org/mailman/listinfo/cellar.
Best Regards,
Dave Rice

= --Apple-Mail=_EC8505EF-8992-48D3-AF40-A232BCBB9B0F-- From nobody Thu Nov 19 07:59:18 2015 Return-Path: X-Original-To: cellar@ietfa.amsl.com Delivered-To: cellar@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0220B1B2C04 for ; Thu, 19 Nov 2015 07:59:10 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.485 X-Spam-Level: X-Spam-Status: No, score=-2.485 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.585] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id an_qZCTMz3r2 for ; Thu, 19 Nov 2015 07:59:04 -0800 (PST) Received: from nostrum.com (raven-v6.nostrum.com [IPv6:2001:470:d:1130::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 92C951B2C07 for ; Thu, 19 Nov 2015 07:59:03 -0800 (PST) Received: from [10.0.1.10] (cpe-70-119-203-4.tx.res.rr.com [70.119.203.4]) (authenticated bits=0) by nostrum.com (8.15.2/8.14.9) with ESMTPSA id tAJFx2wq038350 (version=TLSv1 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 19 Nov 2015 09:59:03 -0600 (CST) (envelope-from ben@nostrum.com) X-Authentication-Warning: raven.nostrum.com: Host cpe-70-119-203-4.tx.res.rr.com [70.119.203.4] claimed to be [10.0.1.10] From: "Ben Campbell" To: cellar@ietf.org Date: Thu, 19 Nov 2015 09:59:02 -0600 Message-ID: <602BE739-954E-4457-A589-06C3AA9A716C@nostrum.com> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.9.3r5180) Archived-At: Subject: [Cellar] CELLAR approved X-BeenThere: cellar@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Codec Encoding for LossLess Archiving and Realtime transmission List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Nov 2015 15:59:10 -0000 Hi, The IESG just approved the CELLAR working group. You should see a formal announcement shortly. Thanks! Ben. From nobody Fri Nov 20 08:01:26 2015 Return-Path: X-Original-To: cellar@ietf.org Delivered-To: cellar@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 041651B3360; Fri, 20 Nov 2015 08:01:21 -0800 (PST) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: The IESG To: "IETF-Announce" X-Test-IDTracker: no X-IETF-IDTracker: 6.10.0 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <20151120160121.5039.80164.idtracker@ietfa.amsl.com> Date: Fri, 20 Nov 2015 08:01:21 -0800 Archived-At: Cc: cellar@ietf.org, The IESG , cellar-chairs@ietf.org Subject: [Cellar] WG Action: Formed Codec Encoding for LossLess Archiving and Realtime transmission (cellar) X-BeenThere: cellar@ietf.org X-Mailman-Version: 2.1.15 List-Id: Codec Encoding for LossLess Archiving and Realtime transmission List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2015 16:01:21 -0000 A new IETF working group has been formed in the Applications and Real-Time Area. For additional information please contact the Area Directors or the WG Chairs. Codec Encoding for LossLess Archiving and Realtime transmission (cellar) ------------------------------------------------ Current Status: Proposed WG Chairs: Tessa Fallon Tim Terriberry Assigned Area Director: Ben Campbell Mailing list Address: cellar@ietf.org To Subscribe: https://www.ietf.org/mailman/listinfo/cellar Archive: https://mailarchive.ietf.org/arch/browse/cellar/ Charter: The preservation of audiovisual materials faces challenges from technological obsolescence, analog media deterioration, and the use of proprietary formats that lack formal open standards. While obsolescence and material degradation are widely addressed, the standardization of open, transparent, self-descriptive, lossless formats remains an important mission to be undertaken by the open source community. FFV1 is a lossless video codec and Matroska is an extensible media container based on EBML (Extensible Binary Meta Language), a binary XML format. There are open source implementations of both formats, and an increasing interest in and support for use of FFV1 and Matroska. However, there are concerns about the sustainability and credibility of existing specifications for the long-term use of these formats. These existing specifications require broader review and formalization in order to encourage widespread adoption. There is also a need for a lossless audio format to complement the lossless video codec and container format. FLAC is a lossless audio codec that has seen widespread adoption in a number of different applications including archival applications. While there are open source implementations of the codec, no formal standards for either the codec itself or its use in container formats currently exist. Review and formalization of the FLAC codec standard and its use in Matroska container formats is needed for wider adoption. Using existing work done by the development communities of Matroska, FFV1, and FLAC, the Working Group will formalize specifications for these open and lossless formats. In order to provide authoritative, standardized specifications for users and developers, the Working Group will seek consensus throughout the process of refining and formalizing these standards. Initial specifications can be accessed here: Specifications: - FFV1: https://mediaarea.net/temp/ffv1.html - Matroska: http://matroska.org/technical/specs/index.html - EBML: http://matroska-org.github.io/libebml/specs.html - FLAC: https://xiph.org/flac/format.html Development Versions: - FFV1: https://github.com/ffmpeg/ffv1 - Matroska: https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml - EBML: https://github.com/Matroska-Org/ebml-specification The Working Group will seek consensus and refinements for specifications for both FFV1 and Matroska in order to provide authoritative, standardized specifications for users and developers. Backward compatibility with existing versions 0-3 of the FFV1 and Matroska specifications will be an important goal, while also reviewing and refining the current version 4 under active development. Although not encouraged, non-backwards-compatible changes to the input specifications will be acceptable if the Working Group determines that the modifications are required to meet the group's technical objectives, provided that the reasons for these changes are clearly documented. Deliverables: - Informational specification for Matroska container format versions 1, 2 and 3 to IESG for publication - Standards Track specification for Matroska container format version 4 to IESG for publication - Informational specification for FFV1 video codec versions 0, 1 and 3 to IESG for publication - Standards Track specification for FFV1 video codec version 4 to IESG for publication - Standards Track specification for FLAC audio codec to IESG for publication Milestones: Apr 2016 - Submit informational specification for Matroska container format versions 1, 2 and 3 to IESG for publication Apr 2016 - Submit informational specification for FFV1 video codec versions 0, 1 and 3 to IESG for publication Jul 2016 - Submit specification for Matroska container format version 4 to IESG (Standards Track) Sep 2016 - Submit specification for FFV1 video codec version 4 to IESG (Standards Track) Dec 2016 - Submit specification for FLAC audio codec to IESG (Standards Track)