Subject: [core] New CoAP Options on the horizon - Accept-Content-Format-Version and Content-Format-Version

From: Alexander Pelov <[email-masked]>
Date: Sun, 9 Apr 2017 16:12:49 +0200

Dear all,

IANA has received a request from Michael Koster to register two new CoAP options:
- Accept-Content-Format-Version
- Content-Format-Version

The registration happens after an expert review and after a couple of mail exchanges and meeting Michael in person in Chicago, we (the experts and Michael) decided that the discussion should be brought to the mailing list, before getting the final approval. 

One of the major questions that needs to be discussed is the following:
What are the reasons *NOT* to allocate a new Content-Format number (RFC 7252) for each new version? (in the HTTP world, "version" is a simple parameter to the Content-Type, so why not treat it as such?)

The question is not trivial, as it touches upon content format negotiation, e.g. how  should the Server use the Accept-Content-Format-Version in order to indicate that it is willing to accept both application/vnd.ocf+cbor AND application/application/vnd.ocf+cbor;version=2.0, with a preference for the latter? In the semantics of the version parameter, the two should be considered completely independent/incompatible types. 

The current way of doing this would be to allocate separate Content-Format numbers to "application/application/vnd.ocf+cbor" (10000 as it stands now) and "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and then use some form of content negotiation. 

This however can have its disadvantages, .. and the way industry does use versioning in some cases (e.g. people DO use the version parameter to version their RESTful APIs.. which means, that NOT having such mechanism in CoAP could theoretically drive them to re-register blocks of Content-Format numbers, just to differentiate between their APIs).

In any case, we need to have this discussion documented on the mailing list (as previously discussed with Michael), so here we go..

Best,
Alexander

---

From: Klaus Hartke <[email-masked]>
Date: Sun, 9 Apr 2017 16:54:39 +0200

> - Accept-Content-Format-Version

Spec?

> One of the major questions that needs to be discussed is the following:

What are the other questions?

> What are the reasons *NOT* to allocate a new Content-Format number (RFC
> 7252) for each new version? (in the HTTP world, "version" is a simple
> parameter to the Content-Type, so why not treat it as such?)

None?

> The question is not trivial, as it touches upon content format negotiation,
> e.g. how  should the Server use the Accept-Content-Format-Version in order
> to indicate that it is willing to accept both application/vnd.ocf+cbor AND
> application/application/vnd.ocf+cbor;version=2.0, with a preference for the
> latter? In the semantics of the version parameter, the two should be
> considered completely independent/incompatible types.

CoAP currently has no features for content format negotiation, only
for the selection of an available content format. If a server wishes
to indicate available content formats, it needs to do so in the
(hypermedia) representation. (The CoAP Accept option can only be used
in requests, not in responses.)

Are the two options a proposal to change that?

Need spec.

> The current way of doing this would be to allocate separate Content-Format
> numbers to "application/application/vnd.ocf+cbor" (10000 as it stands now)
> and "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and
> then use some form of content negotiation.

Yes.

> This however can have its disadvantages, .. and the way industry does use
> versioning in some cases (e.g. people DO use the version parameter to
> version their RESTful APIs.. which means, that NOT having such mechanism in
> CoAP could theoretically drive them to re-register blocks of Content-Format
> numbers, just to differentiate between their APIs).

Working as intended.

> In any case, we need to have this discussion documented on the mailing list
> (as previously discussed with Michael), so here we go..

I'm sure Michael has analyzed the problem, evaluated different options
to solve the problem, and contacted the experts with the best solution
he could find. However, since the identifier spaces in CoAP are so
small and the unrestricted addition of new CoAP features can lead to
fragmentation, every addition must be carefully considered. For that,
it is really helpful to establish an understanding of the problem
before pushing a particular solution.

Klaus

---

From: Alexander Pelov <[email-masked]>
Date: Sun, 9 Apr 2017 22:00:44 +0200

Hi Klaus,

> Le 9 avr. 2017 à 16:54, Klaus Hartke <[email-masked]> a écrit :
> 
> Alexander Pelov wrote:

The other questions are mostly details that we can (easily) figure out once the major one is answered. For example, should the Accept-Content-Format-Version be Repeatable or Not? (I don’t want to get to more details on this for the moment - and they will surely come up if necessary).

> Are the two options a proposal to change that?

Yes, you are right indeed. The *Client* indicates in its CoAP Accept option the Content-Format it is willing to accept. Now, if you have the same Content-Format, but with different versions, then you could imagine having something like: 
Accept: 10000 (which maps to application/application/vnd.ocf+cbor)
Accept-Content-Format-Version: 2.0

Instead of:
Accept: 10001 (application/application/vnd.ocf+cbor;version=2.0 which would need to be registered to 10001 for example)

The intent here is to always indicate "10000" in your hypermedia links, and without changing them to be able to upgrade/downgrade the version of the content-format (e.g. I like to think of this as changing APIs - you upgrade a Client, and then it is on ver2 - but still using the original Content-Format, but this may not be the only reason Michael proposes these options). 

> before pushing a particular solution.

I agree with you on this. Having this understanding - and documented on the mailing list - is the intent of this discussion. 

Alexander

> 
> Klaus
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: "Kovatsch Matthias" <[email-masked]>
Date: Thu, 13 Apr 2017 14:52:40 +0000

Hi all

I want to point out that this is related to the parameter option discussion we had for a long time: https://tools.ietf.org/html/draft-doi-core-parameter-option-03
The work was dropped because there was too much resistance to include this feature (it least in my understanding).

Coming from the design guidelines discussed in T2TRG, it is expected that versioning is handled within the representation format (while a good design also ensures backward and forward compatibility). If a representation format is not compatible with previous versions, it is a different format that needs a new media type, and hence CoAP identifier.
(Note that these guidelines have not reached consensus, and hence cannot be referenced. Please add other views; it might be a valuable T2TRG discussion.)

Ciao
Matthias

> -----Original Message-----
> From: core [mailto:[email-masked]] On Behalf Of Alexander Pelov
> Sent: Sunday, 9 April 2017 22:01
> To: Klaus Hartke <[email-masked]>
> Cc: Core <[email-masked]>
> Subject: Re: [core] New CoAP Options on the horizon - Accept-Content-Format-
> Version and Content-Format-Version
> 
> Hi Klaus,
> 
> 
> >> IANA has received a request from Michael Koster to register two new
> >> CoAP
> >> options:
> The other questions are mostly details that we can (easily) figure out once the
> major one is answered. For example, should the Accept-Content-Format-Version
> be Repeatable or Not? (I don’t want to get to more details on this for the
> moment - and they will surely come up if necessary).
> 
> >
> >> The question is not trivial, as it touches upon content format
> >> negotiation, e.g. how  should the Server use the
> >> Accept-Content-Format-Version in order to indicate that it is willing
> >> to accept both application/vnd.ocf+cbor AND
> >> application/application/vnd.ocf+cbor;version=2.0, with a preference
> >> for the latter? In the semantics of the version parameter, the two should be
> Yes, you are right indeed. The *Client* indicates in its CoAP Accept option the
> Content-Format it is willing to accept. Now, if you have the same Content-
> Format, but with different versions, then you could imagine having something
> like:
> Accept: 10001 (application/application/vnd.ocf+cbor;version=2.0 which would
> need to be registered to 10001 for example)
> 
> The intent here is to always indicate "10000" in your hypermedia links, and
> without changing them to be able to upgrade/downgrade the version of the
> content-format (e.g. I like to think of this as changing APIs - you upgrade a Client,
> and then it is on ver2 - but still using the original Content-Format, but this may
> not be the only reason Michael proposes these options).
> 
> >
> >> The current way of doing this would be to allocate separate
> >> Content-Format numbers to "application/application/vnd.ocf+cbor"
> >> (10000 as it stands now) and
> >> "application/application/vnd.ocf+cbor;version=2.0" (e.g. 10001).. and then
> use some form of content negotiation.
> >
> > Yes.
> >
> >> This however can have its disadvantages, .. and the way industry does
> >> use versioning in some cases (e.g. people DO use the version
> >> parameter to version their RESTful APIs.. which means, that NOT
> >> having such mechanism in CoAP could theoretically drive them to
> >> re-register blocks of Content-Format numbers, just to differentiate between
> their APIs).
> >
> > Working as intended.
> >
> >> In any case, we need to have this discussion documented on the
> >> mailing list (as previously discussed with Michael), so here we go..
> >
> I agree with you on this. Having this understanding - and documented on the
> mailing list - is the intent of this discussion.
> 
> Alexander
> 
> 
> >
> > Klaus
> >

---

From: Michael Koster <[email-masked]>
Date: Thu, 13 Apr 2017 09:48:31 -0700

Hi,

This gets to the core of the problem as I see it.

It is important to be able to construct a request header from the information in a hypermedia control. For links to CoAP resources, we have the content-format number. Even if we have content-format negotiation in CoAP, there is still application awareness of content formats.

The purpose of the content format *version* is to indicate changes in the serialization that don't impact the application compatibility, which the system can handle transparently if it has the correct serializer. 

One use case is to align the representations in OCF with standard IETF content-formats over time, as we are doing with CoRE link-format (RFC6690). Doing this may require a linear update to the OCF content format, which already has a compatible data model, in addition to the implicit support of a standard content format. If there is a move to SenML serialization, for example (hypothetical), but uisng the same OCF data model, then we will need to have an updated OCF content-format version to track OCF versions, even though by then the server may also be able to returnin link-format+cbor (TBD) and senml+cbor (TBD) content format numbers, because of existing OCF applications that are aware of the content-format number.

Thus it is desireable to keep the link attribute value consistent to avoid having to change application layer logic, which is aware of content format numbers and may query a collection using content-format numbers, etc.

Does this make sense?

Best regards,

Michael

> On Apr 9, 2017, at 1:00 PM, Alexander Pelov <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 13 Apr 2017 19:58:19 +0200

I think that this is indicative of a more general problem. Media types have various conventions for indicating subclass relationships: the two-part notation, structured syntax suffixes, but no general way to indicate that a representation is both within a larger and a more specific type. Content-format makes this slightly worse by hiding both the special-case mechanisms cited above and media type parameters, which are another subclassing mechanism.  A fix might be too be able to indicate multiple content-formats. But this requires a bit more thinking. 

Sent from mobile

> On 13. Apr 2017, at 18:48, Michael Koster <[email-masked]> wrote:
> 
> Does this make sense?

---

From: Robert Quattlebaum <[email-masked]>
Date: Thu, 13 Apr 2017 13:22:53 -0700

It is a bit of a long shot, but may be worth bringing up...

Apple came up with a hierarchical content-type mechanism they called the "Universal Type Identifier":

https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html <https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/understanding_utis/understand_utis_conc/understand_utis_conc.html>

This itself doesn't solve the problem, but if there was a hierarchical registry for assigning numeric identifiers at each node (OID style), then the file type could be specified as a sequence of numbers. Applications would then only need to verify that the file type has the appropriate prefix (if well designed, this could just be an opaque string of bytes) and ignore anything they don't understand after.

For example, let's assume the registry has some classes like this:

public.data
0: public.text
0: public.html
1: public.plain-text
0: net.daringfireball.markdown
2: public.json
0: com.example.iot-foo
1: public.image
0: public.jpeg
1: public.tiff

So, for the content type com.example.iot-foo, the type would be encoded as "0.2.0". A TIFF image would be "1.1".

If a browser has a nice JSON pretty-printer but has no idea what com.example.iot-foo is, then it would just check for the "0.2" type prefix, ignoring anything after.

The problem is that the UTI concept never seemed to quite make it to the IETF, or any other standards body (even though it was seemingly designed in such a way as to make standardization somewhat easy). Nonetheless, the concept above might be useful to consider if we are actually thinking about digging up this can of worms.

If we go this route, perhaps some sort of CBOR encoding would be appropriate.

Realistically, this is a big change, if taken as described. I wonder if there is a way to build this concept on top of what we already have?

-- RQ

> On Apr 13, 2017, at 10:58 AM, Carsten Bormann <[email-masked]> wrote:
> 
> On 13. Apr 2017, at 18:48, Michael Koster <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> Does this make sense?

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 19 Apr 2017 15:39:08 +0200

So something like this?

+-- <root>
    |
    +-- A. content coding 'identity'
    |   |
    |   +-- A.0. text/plain; charset=utf-8
    |   |
    |   +-- A.40. application/link-format
    |   |
    |   +-- A.41. application/xml
    |   |   |
    |   |   +-- A.41.65001. application/senml+xml
    |   |   +-- A.41.65002. application/sensml+xml
    |   |
    |   +-- A.42. application/octet-stream
    |   |
    |   +-- A.47. application/exi
    |   |   |
    |   |   +-- A.47.65003. application/senml+exi
    |   |   +-- A.47.65004. application/sensml+exi
    |   |
    |   +-- A.50. application/json
    |   |   |
    |   |   +-- A.50.51. application/json-patch+json
    |   |   +-- A.50.52. application/merge-patch+json
    |   |   +-- A.50.256. application/coap-group+json
    |   |   +-- A.50.11543. application/vnd.oma.lwm2m+json
    |   |   +-- A.50.65005. application/senml+json
    |   |   +-- A.50.65006. application/sensml+json
    |   |
    |   +-- A.60. application/cbor
    |   |   |
    |   |   +-- A.60.16. application/cose; cose-type="cose-encrypt0"
    |   |   +-- A.60.17. application/cose; cose-type="cose-mac0"
    |   |   +-- A.60.18. application/cose; cose-type="cose-sign1"
    |   |   +-- A.60.96. application/cose; cose-type="cose-encrypt"
    |   |   +-- A.60.97. application/cose; cose-type="cose-mac"
    |   |   +-- A.60.98. application/cose; cose-type="cose-sign"
    |   |   +-- A.60.101. application/cose-key
    |   |   +-- A.60.102. application/cose-key-set
    |   |   +-- A.60.10000. application/vnd.ocf+cbor
    |   |   +-- A.60.65007. application/senml+cbor
    |   |   +-- A.60.65008. application/sensml+cbor
    |   |
    |   +-- A.11542. application/vnd.oma.lwm2m+tlv
    |
    +-- B. content coding 'deflate'
    |   |
    |   (same structure as 'identity')
    |
    +-- C. content coding 'gzip'
    |   |
    |   (same structure as 'identity')
    |
    +-- D. content coding 'aes128gcm'
    |   |
    |   (same structure as 'identity')
    |
    +-- E. content coding 'exi'
        |
        +-- E.65003. application/senml+exi
        +-- E.65004. application/sensml+exi

Michael: Does your use case fit in there?

Klaus

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Apr 2017 16:25:39 +0200

Now, is this a hierarchy?

>        +-- E.65003. application/senml+exi

The “senml”-ness could be another node in the tree, with leaves for the cases listed here.

Grüße, Carsten

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 19 Apr 2017 16:50:28 +0200

> Now, is this a hierarchy?

If you want it to be a hierarchy, then there are two options:

+-- <root>
    |
    +-- data model X
    |   |
    |   +-- cbor
    |   +-- exi
    |   +-- json
    |   +-- xml
    |
    +-- data model Y
        |
        +-- cbor
        +-- exi
        +-- json
        +-- xml

+-- <root>
    |
    +-- cbor
    |   |
    |   +-- data model X
    |   +-- data model Y
    |
    +-- exi
    |   |
    |   +-- data model X
    |   +-- data model Y
    |
    +-- json
    |   |
    |   +-- data model X
    |   +-- data model Y
    |
    +-- xml
        |
        +-- data model X
        +-- data model Y

The second allows a processor to fall back to generic
cbor/exi/json/xml if it does not understand data model X/Y. Does the
first have a similar property?

Klaus

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Apr 2017 16:54:17 +0200

> first have a similar property?

If you are about senml, but don’t care about a specific serialization, the other hierarchy would make more sense.

(I’m not sure that this is usefully a hierarchy — I just saw the depictions as a tree and the question was my knee jerk reaction to seeing a tree.)

Grüße, Carsten

---

From: Klaus Hartke <[email-masked]>
Date: Wed, 19 Apr 2017 17:05:32 +0200

+---------------+---------------+---------------+
           /|              /|              /|              /|
          / |             / |             / |             / |
 data    /  |            /  |            /  |            /  |
model Y +---------------+---------------+---------------+   |
        |   |           |   |           |   |           |   |
        |   |           |   |           |   |           |   |
        |   +-----------|---+-----------|---+-----------|---+ deflate
        |  /            |  /            |  /            |  /
        | /             | /             | /             | /
 data   |/              |/              |/              |/
model X +---------------+---------------+---------------+ identity
       cbor            exi             json            xml

Better?

Klaus