Subject: [dispatch] Proposal to add 'haptics' as a new top-level media type - Comments welcome [Was RE: draft-muthusamy-dispatch-haptics-00.txt]

From: Yeshwant Muthusamy <[email-masked]>
Date: Wed, 21 Oct 2020 18:25:11 +0000

Folks,

Re-posting with a more descriptive subjective line, since my previous post (see below) has not engendered any comments (yet).

https://datatracker.ietf.org/doc/draft-muthusamy-dispatch-haptics/

Thanks,
Yeshwant

Yeshwant Muthusamy, Ph.D. | Senior Director, Standards
[email-masked]<mailto:[email-masked]> | +1 469-583-2171

From: Yeshwant Muthusamy
Sent: Tuesday, October 13, 2020 3:53 PM
To: [email-masked]
Cc: Chris Ullrich <[email-masked]>
Subject: draft-muthusamy-dispatch-haptics-00.txt

I would like to initiate discussion on draft-muthusamy-dispatch-haptics:

https://datatracker.ietf.org/doc/draft-muthusamy-dispatch-haptics/

An informal (and earlier) version of this document was discussed on the Media-Types mailing list ([email-masked]<mailto:[email-masked]>):

https://mailarchive.ietf.org/arch/msg/media-types/3cXb69ybEa65oCCIiHjl0MtGXVY/

Look forward to your questions and comments.

Thanks,
Yeshwant

Yeshwant Muthusamy, Ph.D. | Senior Director, Standards
[email-masked]<mailto:[email-masked]> | +1 469-583-2171

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 21 Oct 2020 21:29:59 -0400

In general, this makes sense.  Reading your proposal quickly, I can see
two complications:

One is that in many current usages, haptic data is incorporated into the
same media stream as other media, making it questionable whether haptic
data would be in a stream that could be labeled as such.  I notice that
combined video/audio seems to be common, so this problem is probably
already being solved in that case.

Another is that it seems like few of these media formats are
non-secret.  There are registrations of media formats that are
patent-encumbered, but the specifics of those formats can be
documented.  What would really encourage support for a haptics data type
is one or more open standards.

I would probably be informative to prepare draft IANA registrations of
the haptics type and two or three subtypes and see how writing up the
registrations works out in practice.

Dale

---

From: Yeshwant Muthusamy <[email-masked]>
Date: Fri, 23 Oct 2020 13:38:30 +0000

Dale,

Thanks for the comments. Please see my responses below, prefaced by '[YKM]'.

Regards,
Yeshwant

[email-masked] | +1 469-583-2171

-----Original Message-----
From: Dale R. Worley <[email-masked]> 
Sent: Wednesday, October 21, 2020 8:30 PM
To: Yeshwant Muthusamy <[email-masked]>
Cc: [email-masked]; Chris Ullrich <[email-masked]>
Subject: Re: [dispatch] Proposal to add 'haptics' as a new top-level media type - Comments welcome [Was RE: draft-muthusamy-dispatch-haptics-00.txt]

In general, this makes sense.  Reading your proposal quickly, I can see two complications:

One is that in many current usages, haptic data is incorporated into the same media stream as other media, making it questionable whether haptic data would be in a stream that could be labeled as such.  I notice that combined video/audio seems to be common, so this problem is probably already being solved in that case.
[[YKM]] 'haptics' has been proposed as a first-order media type in the ISOBMFF standard (ISO/IEC 14496-12), at the same level as 'audio' and 'video', so we could indeed have .mp4 files that just have haptic tracks in them. The ISBOMFF Amendment containing our proposal is expected to issue in October 2021 (the January 2022 date in the I-D is an error that will be fixed in the next revision). Bullet #1 in Section 2.1 talks of 'haptics/mp4' files containing just haptics tracks, likely useful for streaming games, haptic files for haptic vests, belts, suits, etc.

Another is that it seems like few of these media formats are non-secret.  There are registrations of media formats that are patent-encumbered, but the specifics of those formats can be documented.  What would really encourage support for a haptics data type is one or more open standards.
[[YKM]] As we mention in Section 2.5 (Haptic Subtypes (in use)), several haptic subtypes like haptics/ahap, haptics/ogg, haptics/ivt are already widely in use, in millions of devices around the world. It is my understanding that we cannot really register a haptics/ahap or haptics/ivt format until the top-level media type ('haptics')  is accepted and registered. Given the standardization progress in MPEG, it seems most logical to use the Standards Track RFC route to get 'haptics' registered as a top-level type first.

I would probably be informative to prepare draft IANA registrations of the haptics type and two or three subtypes and see how writing up the registrations works out in practice.
[[YKM]] I am not sure what you mean by 'draft IANA registrations'. The IANA registration website does not have a provision to specify a new top-level media type. You have to choose from a fixed drop-down list of currently approved top-level types in order to register a new subtype. Given that 'haptics' is not part of the top-level media type list yet, I do not see how I can proceed with a 'draft' registration for any of the subtypes of interest. Am I missing something?

Dale

---

From: John C Klensin <[email-masked]>
Date: Sat, 24 Oct 2020 12:28:56 -0400

--On Friday, October 23, 2020 13:38 +0000 Yeshwant Muthusamy
<[email-masked]> wrote:

> [[YKM]] 'haptics' has been proposed as a first-order media
> type in the ISOBMFF standard (ISO/IEC 14496-12), at the same
> level as 'audio' and 'video', so we could indeed have .mp4
> files that just have haptic tracks in them. The ISBOMFF
> Amendment containing our proposal is expected to issue in
> October 2021 (the January 2022 date in the I-D is an error
> that will be fixed in the next revision). Bullet #1 in Section
> 2.1 talks of 'haptics/mp4' files containing just haptics
> tracks, likely useful for streaming games, haptic files for
> haptic vests, belts, suits, etc.

I think some of the confusion here is that creation of a new
top-level media type is a big deal, both in terms of potential
impact on the Internet and in terms of the number of details
that need to be worked out.  At least one of Dale's comments
(below) is connected to that problem.  So, I would like to
encourage you to think about this a bit differently.

If you are not already in close touch with, and participating
in, ISO/IEC JTC1 SC29 (the body responsible for ISO/IEC
14496-12), make that connection.  If you need help, you might
approach Stephan Wenger, who is IETF's liaison to that SC (see
https://www.ietf.org/about/liaisons/ ).  If they are
establishing the "real" definition for the top-level type and
they (or their MA) are defining or approving the registrations,
ending up with a separate registration system in the IETF -- one
that might not be precisely synchronized -- would be really  bad
news.  

This may suggest a somewhat different approach to what you are
trying to do.  My suggestion (speaking for myself only but with
some small background with IETF Media Type definitions) is that
the ideal way to get the top-level type you are looking for
would start with a request from JTC1 SC29 for this.  The request
should propose a mechanism for ensuring that everything says
synchronized.  One way to do that would be for the authoritative
definitions to be theirs (your I-D goes a small distance in that
direction) and that actual subtyppe registrations use names they
have registered (presumably through their MA) and incorporate
their definitions by reference.

>> documented.  What would really encourage support for a haptics
>> data type is one or more open standards. 

> [[YKM]] As we mention
> in Section 2.5 (Haptic Subtypes (in use)), several haptic
> subtypes like haptics/ahap, haptics/ogg, haptics/ivt are
> already widely in use, in millions of devices around the
> world. It is my understanding that we cannot really register a
> haptics/ahap or haptics/ivt format until the top-level media
> type ('haptics')  is accepted and registered. Given the
> standardization progress in MPEG, it seems most logical to use
> the Standards Track RFC route to get 'haptics' registered as a
> top-level type first.

But this just reinforces the suggestion above.  The IETF cannot
possibly prevent anyone who feels like it from defining a
top-level media type of their own, putting it into use, and
sharing it with their friends or admirers.  In that context, the
purpose of IETF standardization of media types (and IANA
registration of subtypes) is to ensure uniqueness (vastly reduce
the risk of two entities using the same name for different
things) and that there is documentation available and how to
find it.  If "haptics/" with several subtypes is already in
active use, then either there is an existing registration or
assignment system in place on which an IETF definition and
registration procedures should build or an IETF effort should
start with a different name and the understanding that, unless
everyone is on board, the effort could easily build confusion
rather than reducing it.

>> I would probably be informative to prepare draft IANA
>> registrations of the haptics type and two or three subtypes
>> and see how writing up the registrations works out in
>> practice. [[YKM]] I am not sure what you mean by 'draft IANA
>> registrations'. The IANA registration website does not have a
>> provision to specify a new top-level media type. You have to
>> choose from a fixed drop-down list of currently approved
>> top-level types in order to register a new subtype. Given that
>> 'haptics' is not part of the top-level media type list yet, I
>> do not see how I can proceed with a 'draft' registration for
>> any of the subtypes of interest. Am I missing something?

I don't think Dale is but let me say almost the same thing from
a slightly different perspective.  If there is going to be a new
top-level type, your defining document needs to specify exactly
how subtypes will be registered and managed. If there are going
to be submissions to IANA, you should provide a registration
template (or incorporate an existing one by explicit reference)
and I think Dale was asking for examples of that that template
would look like filled in.  You also need to define how
registration applications will be reviewed (see RFC 8126).  And
that brings me back to the suggestion above: especially since
there is probably not broad knowledge of haptics and practices
with them in the IETF, the easiest way to write those sections
(if it is possible) would be to say that me Maintenance Agency
for ISOBMFF (or whatever is actually correct) will register
names and then pass them to IANA for subtype registration,
moving the decision-making out of the IETF's active
responsibilities.

good luck.
   john

---

From: [email-masked] (Dale R. Worley)
Date: Sun, 25 Oct 2020 22:12:49 -0400

Yeshwant Muthusamy <[email-masked]> writes:
> [[YKM]] As we mention in Section 2.5 (Haptic Subtypes (in use)),
> several haptic subtypes like haptics/ahap, haptics/ogg, haptics/ivt
> are already widely in use, in millions of devices around the world. It
> is my understanding that we cannot really register a haptics/ahap or
> haptics/ivt format until the top-level media type ('haptics')  is
> accepted and registered. Given the standardization progress in MPEG,
> it seems most logical to use the Standards Track RFC route to get
> 'haptics' registered as a top-level type first.

I'm sure you are right, but that's not addressing the point I made:  If
the details of an encoding are secret, there is going to be general
antipathy in the IETF toward recognizing it as an "official" media type.
Conversely, pointing to a public document describing the details of the
encodings will improve your chances of getting the concept accepted by
the IETF.

> > I would probably be informative to prepare draft IANA registrations
> > of the haptics type and two or three subtypes and see how writing up
> > the registrations works out in practice.

> [[YKM]] I am not sure what you mean by 'draft IANA registrations'. The
> IANA registration website does not have a provision to specify a new
> top-level media type. You have to choose from a fixed drop-down list
> of currently approved top-level types in order to register a new
> subtype. Given that 'haptics' is not part of the top-level media type
> list yet, I do not see how I can proceed with a 'draft' registration
> for any of the subtypes of interest. Am I missing something?

You understand that you need to write a draft RFC to create the
top-level type, and to get it accepted.  One thing that helps get any
draft RFC accepted is to provide examples of the thing you are
proposing.  In the case of a top-level media type, having the RFC
include registrations of a number of subtypes is useful, because it
shows people that there is practical benefit to be gained immediately,
and also to show them what these subtype registrations look like.
Ideally, these proposed registrations will point to enough information
to enable the reader to implement the encoding.

Dale

---

From: Ned Freed <[email-masked]>
Date: Mon, 26 Oct 2020 07:46:49 -0700 (PDT)

> > accepted and registered.

This may be the case, but it's not the only, or most important case you need to
make when justifying the creation of a top-level type. In particular, you
need to explain why haptics deserve special recognition rather than living
under application, as a myriad of other things (word processing formats,
spreadsheet formats, automative control formats, etc.) do.

I don't think this is a difficult case to make - it seems to me that haptics
data represents a fundamentally different sort of data in the same way that we
differentiate audio, video, etc. But I think section 2 falls short of making
the case currently.

> > Given the standardization progress in MPEG,

> antipathy in the IETF toward recognizing it as an "official" media type.

If by "official" you mean standards tree, you can also replace "antipathy" with
"not going to happen". The specification requirements for standards tree types
(RFC 6838 section 4.10) are quite clear, the consensus behind those
requirements was overwhelming, and I see no reason to believe that has changed.
Indeed, if anything the compromise was to allow specifications for types that
you have to pay to get the specifications.

Of course nothing prevents you from registering a vendor tree type with a name
of the form haptics/vnd.foo. There is no requirement for a publicly available
specification there. And if existing practice must be catered to I suppose you
could make an argument that if a name haptics/foo is already in wide use an
exception could be made to allow that name to be associated with a vendor tree
type. But there is also a good chance that this will be seen as an end run
around the process, and even if succesful, when all is said and done the type
won't be in the standards tree, appearances to the contrary notwithstanding.

> encodings will improve your chances of getting the concept accepted by

While it's possible to register an initial set of media types in a top-level
type proposal, it's definitely not required. And while including illustrative
and noncontroversial examples can be helpful - as noted below - attempting to
resolve every issue you have all at once rarely works well. As such, I
recommend that any potentially problematic registrations be deferred until
after you have the top level type in place.

> to enable the reader to implement the encoding.

Exactly right.

				Ned

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 27 Oct 2020 05:33:06 -0600

This is a very nicely written draft and I think the IETF is the best place to discuss this topic. I hope to see it get agenda time at and upcoming meeting. 

The biggest question I had while reading it was, what do we even do with top level media types at this point ?  What interoperability problem are we trying to solve? Should we deprecate top level media types ? How do we deal with that this media type is often embedded in a video stream?  Is this really different than game controller inputs and moves in real time settings? 

The need for things like the  hstr, hieee, henm, havc all make sense. But I wondering about what we really need the top level type for. Perhaps we should just have a top level type of “other”. I really have no idea of best answer to this but your draft did make me wonder. The flips side is that if top levels don’t mean much, there is no harm in defining a bunch of them. 

> On Oct 21, 2020, at 12:25 PM, Yeshwant Muthusamy <[email-masked]> wrote:
> 
> Folks,
>  
> https://datatracker.ietf.org/doc/draft-muthusamy-dispatch-haptics/ <https://datatracker.ietf.org/doc/draft-muthusamy-dispatch-haptics/>
>  
> Thanks,
> Yeshwant
>  
> [email-masked] <mailto:[email-masked]> | +1 469-583-2171
>  
> From: Yeshwant Muthusamy 
> To: [email-masked] <mailto:[email-masked]>
> Cc: Chris Ullrich <[email-masked] <mailto:[email-masked]>>
> https://datatracker.ietf.org/doc/draft-muthusamy-dispatch-haptics/ <https://datatracker.ietf.org/doc/draft-muthusamy-dispatch-haptics/>
>  
> An informal (and earlier) version of this document was discussed on the Media-Types mailing list ([email-masked] <mailto:[email-masked]>):
>  
> https://mailarchive.ietf.org/arch/msg/media-types/3cXb69ybEa65oCCIiHjl0MtGXVY/ <https://mailarchive.ietf.org/arch/msg/media-types/3cXb69ybEa65oCCIiHjl0MtGXVY/>
>  
> [email-masked] <mailto:[email-masked]> | +1 469-583-2171
>  
> _______________________________________________
> dispatch mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/dispatch <https://www.ietf.org/mailman/listinfo/dispatch>

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 27 Oct 2020 20:43:59 +0900

>> to enable the reader to implement the encoding.

Yes. For a very good example, please look at RFC 8081, which registers 6 
subtypes. In addition to the arguments Ned gave above, including 
non-controversial subtypes in the same document not only shows people 
that there is practical benefit to be gained immediately, the practical 
benefit will actually be gained immediately when the document is accepted.

As an aside, I think delegating the registration of subtypes in a 
certain top-level type to some other standards organization (as alluded 
to by John Klensin) is a bad idea because it would create confusion and 
additional overhead. To have some people familiar with haptics on the 
[email-masked] mailing list in case specific expertise is needed 
would be better.

Regards,   Martin.