Subject: [core] #163: Content-negotiation

From: "core issue tracker" <[email-masked]>
Date: Tue, 05 Jul 2011 08:05:06 -0000

#163: Content-negotiation

 I propose a ticket to add content-type preference indication to coap-07.
 In particular, the proposal is to add a new Elective option called
 "Accept". Here is text lifted from coap-misc-08:

 "The CoAP Accept option would
   when included one or more times in a request, carry one or more media
   types, each of which is an acceptable media type for the client, in
   the order of preference.  If no Accept option is given, the client
   does not express a preference. The client prefers the representation
 returned by the
      server to be in one of the media types, but is willing to accept
      the response also if the server returns a representation in a
      different media type."

 One minor side effect of this change, is that the ct= attribute of core-
 link-format would need to be allowed multiple times. This attribute
 definition should anyways be moved from core-link-format and placed in
 core-coap as it is CoAP specific.

-- 
----------------------------------+-----------------------------------------
 Reporter:  zach@…                |       Owner:  zach@…            
     Type:  protocol enhancement  |      Status:  new               
 Priority:  minor                 |   Milestone:                    
Component:  coap                  |     Version:                    
 Severity:  -                     |    Keywords:                    
----------------------------------+-----------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/core/trac/ticket/163>
core <http://tools.ietf.org/core/>

---

From: Guido Moritz <[email-masked]>
Date: Tue, 5 Jul 2011 17:25:05 +0200

Dear all,

Indeed, I like the idea of some kind of content negotiation. But I existing CoAP draft is very strict in many points about what is allowed by which endpoint and e.g. how a response must be interpreted. 

This proposal instead seems not to be straight forward. A client can include the preferred media type, but the server does not have to care about it?! 

How many media types will be supported by a typical CoAP implementation? For example if a client does not care about it, it will not include any Accept option. But if the client is only able to process XML and the client expresses this in the Accept option, a JSON media type in the response makes no sense. This can be seen as the typical way of negotiating the capabilities between the endpoints. But is the provided media type coupled to this specific resource, to this specific query (e.g. by the query option) or to the complete server? Again, this is the way how HTTP handles this issue, but CoAP much stricter in such points is up to now.

As result, I would suggest to have a specific option whereby a client can express that the Accept media types are preferred, but the client is willing to process also other media types (e.g. by a dedicated media type 'any' or by including an empty media type option as last Accept option). If this 'any' media type is not included, the client strictly requires one of the media types indicated by the Accept options. If no fitting media type is found in the Accept option, the server just responds with a 4.15.

My 2cents,
Guido

> -----Ursprüngliche Nachricht-----
> Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
> core issue tracker
> Gesendet: Dienstag, 5. Juli 2011 10:05
> An: [email-masked]
> Cc: [email-masked]
> Betreff: [core] #163: Content-negotiation
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Zach Shelby <[email-masked]>
Date: Wed, 6 Jul 2011 14:36:35 +0300

Guido,

> This proposal instead seems not to be straight forward. A client can include the preferred media type, but the server does not have to care about it?! 

I would say the philosophy of CoAP is to specify things so as to keep implementations simple, and to allow requests to gracefully succeed with non-critical mismatches (important in allowing minimal implementations). Of course we need to keep resource interactions correct, to do that the server and protocol behavior is pretty strict. Not sure I would describe the interpretation of responses by a client to be strict at all, it is really up to the client what it wants to do with a resource representation.  

> How many media types will be supported by a typical CoAP implementation? For example if a client does not care about it, it will not include any Accept option. But if the client is only able to pro…

The HTTP use case for content-type negotiation is different in my opinion, where both the client and server might support large numbers of media types, and have no (or very little) a-priori knowledge of what the other might support.  

From my experience so far, the CoAP M2M use cases involve servers who offer a representation in a few different pre-determined media types. Clients know these types a priori through standardization (as in the case of ETSI or ZigBee) or via the Link Format. Thus the main intention is to allow a client to show preference for which of those known media types is preferred.  However there may always be mis-matches, for example servers supporting new media-types out of the standard or not updating link descriptions very often.

Making the Accept option elective seems to fit the CoAP model well in my opinion. It allows requests to still succeed even if some servers do not support Accept, and allows requests that don't match to still succeed even if a match isn't found. If the client doesn't find the representation useful, it can just ignore it.  So keep in mind that there are two reasons the client may not get a content-type it put in Accept options, either Accept is not supported by the server, or the server doesn't have any of those preferred content-types.

> As result, I would suggest to have a specific option whereby a client can express that the Accept media types are preferred, but the client is willing to process also other media types (e.g. by a d…

In practice what you are suggesting is to make the Accept option Critical instead of Elective, and treating an empty Accept option as the equivalent of HTTP Accept */*. That would of course mean that servers not supporting Accept would fail, requiring a second request without Accept. Is that any more efficient than possibly receiving a representation the client might have to throw away on the first shot?

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Guido Moritz <[email-masked]>
Date: Wed, 6 Jul 2011 14:32:17 +0200

I would not agree that servers are likely to support several media types in
parallel. But I totally agree that it is in the responsibility of further
standardization bodies to define domain specific 'profiles', including a
limited number of media types.

But in a M2M scenario, why should a client prefer any media type? Based our
measurements, even on 8bit platforms, parsing either XML, EXI, JSON our
whatever has never been the bottleneck by means of timing. It is always
about the required ROM/RAM usage of the parsers. So if I can understand more
than one media type, why should I prefer one? I don't have a use case for
this. I would say it is more important to indicate that the media types in
the Accept are not preferred, but the only ones the client can parse at all.

To relax this, in coap-misc two Accept options are defined, one elective and
one critical. Maybe we can have both?

Guido

> Von: Zach Shelby [mailto:[email-masked]]
> Gesendet: Mittwoch, 6. Juli 2011 13:37
> An: Guido Moritz
> Cc: [email-masked]
> Betreff: Re: [core] #163: Content-negotiation
> 
> Guido,
> 
> > Indeed, I like the idea of some kind of content negotiation. But I
existing CoAP
> draft is very strict in many points about what is allowed by which
endpoint and
> e.g. how a response must be interpreted.
> >
> > This proposal instead seems not to be straight forward. A client can
include
> the preferred media type, but the server does not have to care about it?!
> 
> I would say the philosophy of CoAP is to specify things so as to keep
> implementations simple, and to allow requests to gracefully succeed with
non-
> critical mismatches (important in allowing minimal implementations). Of
> course we need to keep resource interactions correct, to do that the
server and
> protocol behavior is pretty strict. Not sure I would describe the
interpretation of
> responses by a client to be strict at all, it is really up to the client
what it wants
> to do with a resource representation.
> 
> > How many media types will be supported by a typical CoAP implementation?
> For example if a client does not care about it, it will not include any
Accept
> option. But if the client is only able to process XML and the client
expresses this
> in the Accept option, a JSON media type in the response makes no sense.
This
> can be seen as the typical way of negotiating the capabilities between the
> endpoints. But is the provided media type coupled to this specific
resource, to
> this specific query (e.g. by the query option) or to the complete server?
Again,
> this is the way how HTTP handles this issue, but CoAP much stricter in
such
> points is up to now.
> 
> The HTTP use case for content-type negotiation is different in my opinion,
> where both the client and server might support large numbers of media
types,
> and have no (or very little) a-priori knowledge of what the other might
support.
> 
> From my experience so far, the CoAP M2M use cases involve servers who
offer
> a representation in a few different pre-determined media types. Clients
know
> these types a priori through standardization (as in the case of ETSI or
ZigBee) or
> via the Link Format. Thus the main intention is to allow a client to show
> preference for which of those known media types is preferred.  However
there
> may always be mis-matches, for example servers supporting new media-types
> out of the standard or not updating link descriptions very often.
> 
> Making the Accept option elective seems to fit the CoAP model well in my
> opinion. It allows requests to still succeed even if some servers do not
support
> Accept, and allows requests that don't match to still succeed even if a
match
> isn't found. If the client doesn't find the representation useful, it can
just ignore
> it.  So keep in mind that there are two reasons the client may not get a
content-
> type it put in Accept options, either Accept is not supported by the
server, or
> the server doesn't have any of those preferred content-types.
> 
> > As result, I would suggest to have a specific option whereby a client
can
> express that the Accept media types are preferred, but the client is
willing to
> process also other media types (e.g. by a dedicated media type 'any' or by
> including an empty media type option as last Accept option). If this 'any'
media
> type is not included, the client strictly requires one of the media types
indicated
> by the Accept options. If no fitting media type is found in the Accept
option, the
> server just responds with a 4.15.
> 
> In practice what you are suggesting is to make the Accept option Critical
> instead of Elective, and treating an empty Accept option as the equivalent
of
> HTTP Accept */*. That would of course mean that servers not supporting
> Accept would fail, requiring a second request without Accept. Is that any
more
> efficient than possibly receiving a representation the client might have
to throw
> away on the first shot?

---

From: Zach Shelby <[email-masked]>
Date: Thu, 7 Jul 2011 13:42:05 +0300

> limited number of media types.

Selected resources on servers will support several media types, we already have seen that need from two SDOs that are designing for the use of CoAP. They are defining which formats those resources need to support in the standard, but require the Accept feature for a client to indicate which one is preferred. This is by far the most likely use case where CoAP Accept will really be used. 

> than one media type, why should I prefer one?

The server may offer three representations, say XML, EXI and JSON. You may only understand XML and JSON. You get the point. 

> I don't have a use case for
> this.

Then why do you care? (kidding). The preference approach seems to work fine for the use cases in mind. 

> I would say it is more important to indicate that the media types in

In the end, what does that really change? The only difference is that you will get a useless representation in the "preferred" model, or a 4.06 (not acceptable) in the "only accept" case. They achieve the same thing in reality. 

> one critical. Maybe we can have both?

I thought about that, but it is getting needlessly complex (and confusing) to have two different options for Accept.

From the viewpoint of efficiency and simplicity, I would definitely prefer for this to be Elective. Now, there is nothing stopping us from making this "only accept" and returning a 4.06 error code even if the option is elective. However, resource on servers that don't support Accept (and thus only have a single media type) will just ignore it and return the representation it has, and the client would just have to deal with it. Does that behavior make sense to people? 

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 7 Jul 2011 14:04:48 +0200

> From the viewpoint of efficiency and simplicity, I would definitely prefer for this to be Elective. Now, there is nothing stopping us from making this "only accept" and returning a 4.06 error code…

Well, there are two semantics.

-- I need this (one out of a set of) content types, and I'm not interested in anything else.
	If the option is Elective, the client still cannot rely on getting just that.  So for this semantics, the option should be Critical.

-- Please give my this (one out of a set of) content type, but I'd be happy with anything else.
	If the option is Critical, the request might be needlessly rejected.  So for this semantics, the option should be Elective.

Which of the ones do we need?  Since I consider the cost of getting an unwanted answer (as opposed to an error answer) negligible*), there is little value to the first semantics.  There is even less value to an implementation that would honor a Critical Accept and choose to ignore an Elective one (why on earth would one do that?).  That's why I was happy with only having the second.  Maybe it would help to add some implementer advice that, to maximize interoperability, the Accept *should* be honored if the implementation can do that (well, whatever this advice will be, it will be wishy-washy, so it can't be a SHOULD).

Gruesse, Carsten

*) Tell me if you have a counter-example that is not hypothetical.

---

From: Guido Moritz <[email-masked]>
Date: Thu, 7 Jul 2011 14:30:31 +0200

Maybe I am repeating myself, sorry!

I don't get the point why on earth an endpoint (in an M2M) would prefer to
use a specific media type. Either it is capable of parsing it or it is not.
Or do we have implementations which are clever enough to use e.g. XML for
payload <100Byte and for payload >100Byte switching to EXI?

If it is in the decision of the server to use the preferred media type or to
use anything else, can a client rely on that the next request will result in
the same (not wanted) media type? Or does a server deciding this each time
different?

Guido

> Von: Carsten Bormann [mailto:[email-masked]]
> Gesendet: Donnerstag, 7. Juli 2011 14:05
> An: Zach Shelby
> Cc: Guido Moritz; [email-masked]
> > From the viewpoint of efficiency and simplicity, I would definitely
prefer for
> this to be Elective. Now, there is nothing stopping us from making this
"only
> accept" and returning a 4.06 error code even if the option is elective.
However,
> resource on servers that don't support Accept (and thus only have a single
> media type) will just ignore it and return the representation it has, and
the
> client would just have to deal with it. Does that behavior make sense to
people?
> -- I need this (one out of a set of) content types, and I'm not interested
in
> anything else.
> 	If the option is Elective, the client still cannot rely on getting
just that.
> So for this semantics, the option should be Critical.
> 
> -- Please give my this (one out of a set of) content type, but I'd be
happy with
> anything else.
> 	If the option is Critical, the request might be needlessly rejected.
So for
> this semantics, the option should be Elective.
> 
> Which of the ones do we need?  Since I consider the cost of getting an
> unwanted answer (as opposed to an error answer) negligible*), there is
little
> value to the first semantics.  There is even less value to an
implementation that
> would honor a Critical Accept and choose to ignore an Elective one (why on
> earth would one do that?).  That's why I was happy with only having the
second.
> Maybe it would help to add some implementer advice that, to maximize
> interoperability, the Accept *should* be honored if the implementation can
do
> that (well, whatever this advice will be, it will be wishy-washy, so it
can't be a
> SHOULD).

---

From: Zach Shelby <[email-masked]>
Date: Thu, 7 Jul 2011 15:40:41 +0300

> Which of the ones do we need?  Since I consider the cost of getting an unwanted answer (as opposed to an error answer) negligible*), there is little value to the first semantics.  There is even les…

I came to the same conclusion. 

Now I took a shot to improve the Accept text and made an attempt to include SHOULD language as you suggest. Here is the current Accept section text from the SVN version of CoAP. Does this need improvement anyone?

5.10.5.  Accept

   The CoAP Accept option indicates when included one or more times in a
   request, carry one or more media types, each of which is an
   acceptable media type for the client, in the order of preference.
   The representation format is given as a numeric media type identifier
   that is defined in the CoAP Media Type registry (Section 11.3).  If
   no Accept options are given, the client does not express a preference
   (thus no default value is assumed).  The client prefers the
   representation returned by the server to be in one of the media types
   indicated.  The server SHOULD return one of the preferred media types
   if available.  As a server might not support the Accept option (and
   thus would ignore it as it is elective), or a server might not have a
   preferred media type available, the client needs to be prepared to
   receive a representation in a different media type.  The client can
   simply discard a representation it can not make use of.

   This option is "elective".  It MAY occur more than once.

---

From: Guido Moritz <[email-masked]>
Date: Thu, 7 Jul 2011 16:15:28 +0200

I would still love to have more detailed information for the client how the
interpret a not wanted response. Otherwise it would be hard for a client to
determine which the real capabilities of the server are and which just have
occurred, e.g., due to implementation or situation specific side effect
(e.g. RAM overload, temporary low throughput...).

Something like: "If the server is temporarily not able to provide a media
type requested in the Accept option, the server SHOULD respond with a
different media type. If the server is entirely not capable of providing the
requested media type, the server SHOULD respond with a 4.06."

This is already possible with the existing text, but I guess it would be
helpful to have a more precise definition. I am not sure which side effects
such a behavior might have.

And I don't want to overstrain this discussion :)

Guido

> Gesendet: Donnerstag, 7. Juli 2011 14:41
> An: Carsten Bormann
> > -- I need this (one out of a set of) content types, and I'm not
interested in
just that.
happy with
So for
little
implementation that
second.
do
can't be a
> Now I took a shot to improve the Accept text and made an attempt to
include
> SHOULD language as you suggest. Here is the current Accept section text
from
> the SVN version of CoAP. Does this need improvement anyone?
> 
> 5.10.5.  Accept
>

---

From: "core issue tracker" <[email-masked]>
Date: Fri, 08 Jul 2011 11:17:13 -0000

Changes (by zach@…):

  * status:  new => closed
  * resolution:  => fixed

Comment:

 Closing this ticket for the -07 release, acknowledging that the text will
 still need some tweaking.

-- 
----------------------------------+-----------------------------------------
 Reporter:  zach@…                |        Owner:  zach@…            
     Type:  protocol enhancement  |       Status:  closed            
 Priority:  minor                 |    Milestone:                    
Component:  coap                  |      Version:                    
 Severity:  -                     |   Resolution:  fixed             
 Keywords:                        |  
----------------------------------+-----------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/core/trac/ticket/163#comment:1>
core <http://tools.ietf.org/core/>

---

From: "Dijk, Esko" <[email-masked]>
Date: Sun, 10 Jul 2011 14:54:19 +0200

I would agree with Zach that having one Elective Accept-option seems right for the described use case (i.e. SDO/ZigBee/ETSI multiple content formats).

[I'm probably repeating Zach's points here:]
1- Elective enables minimal implementations to be simpler (ignore "Accept" entirely; return fixed content type)
2- a CoAP client according to the use case knows what a server offers, and whether it supports Accept, since it's specified in a standard and/or discovered via Link Format. So the client would request from these formats and almost always the server is able to serve one of these.

Suppose the server in case 2 still returns something the client doesn't understand, the client may interpret this as a form of "server error 4.15". Note, I think that even with an Elective Accept option a server can still legally return a 4.15 - this may even be agreed so by the people of the related standard.
Finally if RAM or resources are the problem the server should respond 5.00 or 5.03, isn't it?

Esko

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Guido Moritz
Sent: Thursday 7 July 2011 16:15
To: 'Zach Shelby'; 'Carsten Bormann'
Cc: [email-masked]
Subject: Re: [core] #163: Content-negotiation

Guido

interested in
just that.
happy with
So for
little
implementation that
second.
do
can't be a
include
from

The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: "Kovatsch Matthias" <[email-masked]>
Date: Mon, 18 Jul 2011 19:27:06 +0000

Hi

I agree that we need an (elective) Accept option to choose between different representations.
But why should CoAP allow multiple occurrences of that option?
Here, I totally agree with Guido: Why should a client prefer one over another? It just complicates the implementation on both sides.
The representations to choose from should be announced in the Link Format. If this is not done properly at the server-side, the client can still fall back to a trial-and-error behavior to check of one of his supported formats is available (there won't be that many..).

Best regards
Matthias

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Dijk, Esko
> Sent: Sonntag, 10. Juli 2011 14:54
> To: Guido Moritz; 'Zach Shelby'; 'Carsten Bormann'
> Cc: [email-masked]
> I would agree with Zach that having one Elective Accept-option seems right
> for the described use case (i.e. SDO/ZigBee/ETSI multiple content
> formats).
> 
> 2- a CoAP client according to the use case knows what a server offers, and
> whether it supports Accept, since it's specified in a standard and/or
> discovered via Link Format. So the client would request from these formats
> and almost always the server is able to serve one of these.
> 
> Suppose the server in case 2 still returns something the client doesn't
> understand, the client may interpret this as a form of "server error
> 4.15". Note, I think that even with an Elective Accept option a server can
> still legally return a 4.15 - this may even be agreed so by the people of
> the related standard.
> I would still love to have more detailed information for the client how
> the interpret a not wanted response. Otherwise it would be hard for a
> client to determine which the real capabilities of the server are and
> which just have occurred, e.g., due to implementation or situation
> specific side effect (e.g. RAM overload, temporary low throughput...).
> 
> different media type. If the server is entirely not capable of providing
> the requested media type, the server SHOULD respond with a 4.06."
> 
> helpful to have a more precise definition. I am not sure which side
> effects such a behavior might have.
> 
> > would honor a Critical Accept and choose to ignore an Elective one
> > (why on earth would one do that?).  That's why I was happy with only
> > having the
> second.
> The information contained in this message may be confidential and legally
> protected under applicable law. The message is intended solely for the
> addressee(s). If you are not the intended recipient, you are hereby
> notified that any use, forwarding, dissemination, or reproduction of this
> message is strictly prohibited and may be unlawful. If you are not the
> intended recipient, please contact the sender by return e-mail and destroy
> all copies of the original message.
>