Subject: Re: [core] EXI content type and encoding (was: Resolution to tickets #176, #177, #181)

From: Guido Moritz <[email-masked]>
Date: Fri, 17 Feb 2012 13:43:39 +0100

I have to point out that I got lost somewhere in this discussion about the
EXI content type and encoding. But because I am working with EXI and
implementing it for Contiki+TelosB, maybe I could clarify a bit. 

(Please take care of the used wordings: serialization, encoding and
representing)
>From a more general data representation perspective, EXI and Unicode XML
(e.g. utf-8) are different encodings, which are both capable of representing
the same serialized structured data objects. Native Unicode XML can have
different encodings like e.g. utf-8 and utf-16. The EXI Format is a
completely different encoding format. 
EXI has much more options, leading to very different meanings about what a
single bit or byte on a wire stands for. EXI e.g. differentiates between
schema-informed, schema-less, strict, non-strict, bit-aligned, byte-aligned
and compressed modes and their combination (just to name few of the relevant
buzz words). Hence, the EXI format has its very own mechanisms within an EXI
stream to indicate the encoding mode and options to be used to parse each
single stream.

Summing up my suggestion would be:
1) EXI is a very important format for constrained environments and must be
considered
2) Keep application/exi media type as it is
3) Do not add any further content encoding options for the EXI format, as
such indication mechanism are already part of the EXI format itself.
Defining such additional options for e.g. CoAP or HTTP is out of scope for
the CoRE WG. If required, it MUST be part of the EXI standardization process
within the W3C to define the option values and their semantics in the
context of the EXI format.

Sorry if I did not got the point of the discussion, 
Guido

> -----Ursprüngliche Nachricht-----
> Von: [email-masked] [mailto:[email-masked]] Im Auftrag von
> Carsten Bormann
> Gesendet: Donnerstag, 16. Februar 2012 23:10
> An: Peter Saint-Andre
> Cc: [email-masked] WG
> Betreff: Re: [core] Resolution to tickets #176, #177, #181
> 
> On Feb 16 2012, at 22:51, Peter Saint-Andre wrote:
> 
> > <hat type='individual'/>
> >
> > On 2/15/12 8:23 AM, Carsten Bormann wrote:
> >
> >> As a result, we are proposing the body of coap-misc as the resolution
> >> for tickets #176, #177 (and indirectly for #174), and #181, as well
> >> as the vendor-defined option that was discussed.
> >
> > Regarding #181, would the proposed change obviate the need for the
> > +exi
> > prefix as discussed on the apps-discuss list (since CoAP entities
> > could
> > handle content-encoding)?
> 
> I think that discussion is slightly confused (I haven't tried to
> intervene, as I'm not an EXI person).
> 
> application/foo+exi could be a media type for a schema-informed EXI
> encoding of foo+xml.
> Content-Encoding is meant to be futzed with in proxies (all the 214
> verbage notwithstanding), so from my point of view needs to be
> schemaless.
> 
> But there are a lot of angels and pinheads in the interpretation of
> these attributes and concepts, so I think we need to be able to cope
> with both outcomes.
> 
> The Content-Encoding column is mainly an insurance against people
> claiming we need a Content-Encoding Option in CoAP because we can't
> have encoded media types.
> HTTP wants media type and Content-Encoding to be orthogonal and freely
> combinable.
> Since constrained devices are unlikely to have that level of
> orthogonality, I'd rather make it obvious that both media type and
> Content-Encoding are tied together in your decision of what to ship
> over the wireless.
> 
> If the "you can't have +exi but must encode that aspect of the
> representation format in Content-Encoding" faction wins, there will be
> zero change for us, except in the documents where we replace "+exi --"
> with "+xml exi" in some tables.
> 
> Protocol design 101: Design so you win with either possible outcome of
> a tussle.
> 
> Gruesse, Carsten
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Feb 2012 14:11:12 +0100

> 2) Keep application/exi media type as it is

That is the point where the discussion begins:
application/exi tells you about the basic syntax, but not about the interpretation of the data.
There are about three base syntaxes of interest right now: (character) XML, EXI, and JSON.
But we also need higher level format definitions such as senml, which in turn can use (character) XML, EXI, JSON.
So how do we encode the combination?

SenML says:

   6.  JSON Representation (application/senml+json) . . . . . . . . .  7
   7.  XML Representation (application/senml+xml) . . . . . . . . . . 11
   8.  EXI Representation (application/senml+exi) . . . . . . . . . . 12

Works for me.

But then there are purists that think EXI is a Content-Encoding of XML (while character XML is "unencoded" XML).
So in HTTP you would have:

Content-Type: application/senml+xml
Content-Encoding: exi

While the other two would be:

Content-Type: application/senml+json

and

Content-Type: application/senml+xml

To isolate us from this discussion, I propose to combine the media type and the Content-Encoding in our numeric Content-Type option.

Grüße, Carsten

---

From: Guido Moritz <[email-masked]>
Date: Fri, 17 Feb 2012 14:34:30 +0100

> application/exi tells you about the basic syntax, but not about the
interpretation
> of the data.
> There are about three base syntaxes of interest right now: (character)
XML,
> EXI, and JSON.
> But we also need higher level format definitions such as senml, which in
turn
> can use (character) XML, EXI, JSON.

Works this way also for HTTP as far as I know and also for other
applications specific formats (e.g. SOAP ;).
application/soap+xml
application/soap+exi

> But then there are purists that think EXI is a Content-Encoding of XML
(while
> character XML is "unencoded" XML).

Wrong! As stated in my last mail, EXI and Unicode (you say character) XML
are different encodings for one and the same serialized data objects. JSON
and Fast Infoset are the next two candidates. All these formats are not
generated from the Unicode (character) XML, but instead in parallel to
Unicode (character) XML. The purists should keep in mind that 'XML' is a
quite verbose standards family and not only the single encoding format they
have in mind.

> To isolate us from this discussion, I propose to combine the media type
and the
> Content-Encoding in our numeric Content-Type option.

+1. Maybe still requires careful attention for each dedicated format. 

Guido

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Feb 2012 15:24:45 +0100

> application/soap+exi

Yeah, except that the first one exists and the other one doesn't:

http://www.iana.org/assignments/media-types/application/index.html

> Wrong!

You and I agree on that…
But there is still considerable confusion in the wider community.

> As stated in my last mail, EXI and Unicode (you say character) XML
> and Fast Infoset

Yep:

application/soap+fastinfoset	[ITU-T ASN.1 Rapporteur]

> are the next two candidates.

(Except that JSON has a different abstract syntax, too.)

> All these formats are not

(In SGML, we called that a "concrete syntax".  ASN.1 has encoding rules.  Etc.)

> +1. Maybe still requires careful attention for each dedicated format. 

I think the two of us agree that viewing EXI as a Content-Encoding of (character) XML is problematic.

Grüße, Carsten

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 17 Feb 2012 08:43:07 -0700

<hat type='individual'/>

On 2/17/12 7:24 AM, Carsten Bormann wrote:
>> application/soap+xml application/soap+exi
> 

I am not an EXI expert, but my understanding from a conversation with
Zach in Taipei (and afterward on the apps-discuss list) is that one
could see EXI as either (1) a content-encoding of XML or (2) a new and
different content-type. The matter isn't settled, but it would be good
to settle it. :)

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Zach Shelby <[email-masked]>
Date: Fri, 17 Feb 2012 18:25:29 +0200

On Feb 17, 2012, at 5:43 PM, Peter Saint-Andre wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> <hat type='individual'/>
> 

And regarding the +exi registration requirements I have an AP to write up a -00 draft on that. Will forward that also to this WG.

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Guido Moritz <[email-masked]>
Date: Fri, 17 Feb 2012 21:14:48 +0100

> different content-type. The matter isn't settled, but it would be good

I also had to think about this issue for a while, because I have to explain the differences and perspectives as part of my Phd thesis...keep fingers cross to have finished it before Paris ;). Overall I concluded in (2) and describe EXI as a completely new and different 'content-type', i.e. 'on wire encoding' of XML-Infoset serialized object structures.

I think how and why people favor one of the two options is how they use it. There are applications/implementations that are already using XML and to save bandwidth now the data stream is changed for several parts/communications into the EXI format. In this use case, EXI can even be a compressor, i.e. compression is the process changing the data representation to a more efficient one. But because their applications/implementations are still parsing and processing native Unicode XML, EXI is just another encoding for the existing XML structures. From this perspective absolutely feasibly to favor (1). 

But EXI is, in contrast to e.g. gzip, also capable to be a standalone representation that can be parsed and processed directly by the implementations. This is how the EXI format was designed by the W3C, as a complete replacement for Unicode XML if required. In this case there is no need to change EXI to XML to process it. In this case (2) is more feasible. And this is also the way we are using EXI within our 6LoWPAN-related implementations. 

My 2 cents,
Maybe we can discuss it in Paris while having a nice wine, 
Guido