Subject: [core] Designs to resolve streaming issues in SenML

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Wed, 13 Jan 2016 13:57:02 +0000

The changes we made to SenML to support streaming in the -02 version raised many issues for lots of people. In the end, one of the biggest problems is it forces all things receiving SenML to support streaming. 

I have an alternative design for streaming that I think addresses the issues raised and I would like the WG to think about two alternative designs for SenML and give me some direction on which one to use. 

The first design is the current -02 design with simple fixes and is in draft draft-jennings-core-senml-03

The second design is the new design I am proposing and is in draft-jennings-core-senml-04. It many way it moves back to be much closer to what was in -01 than what is in -02. 

The drafts can be found at 

https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/

and 

https://datatracker.ietf.org/doc/draft-jennings-core-senml/04/

Practically speaking the differences are not that much between the two but they do resolve a bunch of the issue. Let me discuss this in terms of the JSON representation and the rest of the representations mirror that.  In the new design measurements are put in JSON object as always. The object are put in array to represent a series of measurements. The objects can also have the base values right in the object and the base values apply to all the things in the object and any future objects in the array. 

Here is an example of something that is not a stream in both old and new design.

OLD 

[{"bn": "urn:dev:mac:0024befffe804ff1/"},
 [ { "n": "voltage", "t": 0, "u": "V", "v": 120.1 },
   { "n": "current", "t": 0, "u": "A", "v": 1.2 } ]
]

NEW

[{"bn": "urn:dev:mac:0024befffe804ff1/"},
 { "n": "voltage", "t": 0, "u": "V", "v": 120.1 },
 { "n": "current", "t": 0, "u": "A", "v": 1.2 } 
]

Note the old design has an array with a base object, followed by a second array of measurements. The new design just has an single array and the first object happens to only have base values that apply to future object in the array. 

This could also be sent as

[
    {
	"t": 0, 
      	"n": "voltage", 
	"u": "V", 
     	"bn": "urn:dev:mac:0024befffe804ff1/",
	"v": 120.1
   },
   { 
	"n": "current", 
	"t": 0, 
	"u": "A", 
	"v": 1.2 
   } 
]

Here is an example of streaming with old and new design 

OLD 

[{"bn": "http://[2001:db8::1]",
  "bt": 1320067464,
  "bu": "%RH"},
 [ { "v": 21.2, "t": 0 },
   { "v": 21.3, "t": 10 },
   { "v": 21.4, "t": 20 },
   { "v": 21.4, "t": 30 },
   { "v": 21.5, "t": 40 },
   { "v": 21.5, "t": 50 },
   { "v": 21.5, "t": 60 },
   { "v": 21.6, "t": 70 },
   { "v": 21.7, "t": 80 },
   { "v": 21.5, "t": 90 },
...

NEW 

[ {"bn": "http://[2001:db8::1]",
  "bt": 1320067464,
  "bu": "%RH"},
   { "v": 21.2, "t": 0 },
   { "v": 21.3, "t": 10 },
   { "v": 21.4, "t": 20 },
   { "v": 21.4, "t": 30 },
   { "v": 21.5, "t": 40 },
   { "v": 21.5, "t": 50 },
   { "v": 21.5, "t": 60 },
   { "v": 21.6, "t": 70 },
   { "v": 21.7, "t": 80 },
   { "v": 21.5, "t": 90 },
...

The draft does not have it yet but an applications that supports receiving streaming is written is a substantially different way that one that does not. I think the best way to deal with this is simply use a different mine type for streaming ( calls it senml_stream for now but send me suggestions on name for it). That way it can be clearly indicated and negotiated in existing transport protocols. 

Appreciate any thoughts people have on pro / cons of the -03 design vs -04 design.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Jan 2016 15:30:37 +0100

Cullen Jennings (fluffy) wrote:
> pro / cons of the -03 design vs -04 design.

One comment I made when this was discussed in Yokohama was that the -04
design makes the processor examine every single measurement record
whether there is a change to the base values (and, if there are only
changes to base values so that record isn't really a measurement
record).  That might impede high speed processing a bit.  E.g., it could
become harder to merge two time sequences because each single entry in
them might be changing a base value that is relevant to entries in the
other stream.  But I haven't implemented either model, so that is just a
hunch.

Apart from that kind of consideration, that part of the approach -04 is
pretty much equivalent to -03.

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Thu, 14 Jan 2016 18:43:38 -0800

Hi,

I have been working with JSON serializations of senml and link data lately and I have some comments on the new SenML draft. I haven’t kept up with the format changes because I have something already working that I like that is based on the senml core wg draft-01. 

There are two problems with the new formats (-02 and -04) for my use case. The first is that I have added more element types to extend senml to represent collections that may have links, forms, and data items. The second is the issue of mapping vs. streaming as discussed previously.  First, I’ll try to explain the issue as I see it with mapping vs. streaming.

This is how the “collection” example appears in section 6.1.4 of draft-04:

   [{"bn": "http://[2001:db8::2]/",
     "bt": 1320078429,
     "ver": 3},
      { "n": "temperature", "v": 27.2, "u": "Cel" },
      { "n": "humidity", "v": 80, "u": "%RH" }
   ]

The block formatting of the first map and addition of whitespace to the second 2 maps gives the visual impression that there is some hierarchy. The reformatted example below using standard json pretty print makes it easier to see how this parses into a data structure using standard json tools. I’d recommend following this format to make the examples consistent and easy to read. I will do the same in my drafts.

[
  {
    "bn": "http://[2001:db8::2]/",
    "bt": 1320078429,
    "ver": 3
  },
  {
    "n": "temperature",
    "u": "Cel",
    "v": 27.2
  },
  {
    "n": "humidity",
    "u": "%RH",
    "v": 80
  }
]
As pointed out, the streaming format can only be processed in streaming form. Storing this in memory using a JSON parser does not result in a structure that can be easily sorted or indexed by base element. If I were to design a parser for this, it would have a state machine that would consume the top level array elements in sequence and create an in-memory structure that looks like the structures described in draft-02:

[
  {
    "bn": "http://[2001:db8::2]/",
    "bt": 1320078429,
    "ver": 1
  },
  [
    {
      "n": "temperature",
      "u": "Cel",
      "v": 27.2
    },
    {
      "n": "humidity",
      "u": "%RH",
      "v": 80
    }
  ]
]
Or draft-01:

{
  "bn": "http://[2001:db8::2]/",
  "bt": 1320078429,
  "ver": 1
  "e": [
    {
      "n": "temperature",
      "u": "Cel",
      "v": 27.2
    },
    {
      "n": "humidity",
      "u": "%RH",
      "v": 80
    }
  ],
}
With these earlier structures I could use standard JSON parse tools and then selection algorithms that match the “bn” value and select the contents within that item’s scope (for patching, etc.). With the streaming formats I need to add another level of serialize/deserialize between the content format and the actual program data model and process elements in order.

IOW, I already have a JSON parser and a JSON data structure filter that work together. Now with this format change, I need to add another layer of deserializing to create an index or remap the base elements.

The other issue is that in draft-01 the element tag “e” maps to the XML serialization <e/> tags nicely. I have tools that process both XML and JSON that understand this. Also, this is easily extensible to add more element classes to the senml data model, which I have done of links and forms.

{
  "bn": "/",
  "e": [
    {
      "n": "test",
      "sv": "testValue"
    }
  ],
  "l": [
    {
      "href": "",
      "rel": [
        "self"
      ]
    },
    {
      "href": "test",
      "rel": "sub"
    }
  ]
}

I have indicated this with a content format “collection+senml+json”, to indicate the senml data model with a collection extension or subtype.

Since the recent changes in senml, I have considered forking -01 and creating a new set of media types called hyper-senml, or “hsml”, so “application/hsml+json”, “application/hsml+xml”, “application/collection+hsml+json”, etc.

I guess my preferred outcome would be a senml optimized for mapping that looks like -01 and can be extended to add element classes, and a separate, streaming optimized format.

Best regards,

Michael

> On Jan 13, 2016, at 5:57 AM, Cullen Jennings (fluffy) <[email-masked]> wrote:
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Michael Koster <[email-masked]>
Date: Thu, 14 Jan 2016 19:04:15 -0800

Note: A reference implementation (in progress) of the Hypermedia Collection described in https://datatracker.ietf.org/doc/draft-ietf-core-interfaces/ <https://datatracker.ietf.org/doc/draft-ietf-core-interfaces/> uses the collection+senml content type mentioned below and is available at https://github.com/mjkoster/MachineHypermediaToolkit <https://github.com/mjkoster/MachineHypermediaToolkit> for an example of the use case.

> On Jan 14, 2016, at 6:43 PM, Michael Koster <[email-masked]> wrote:
> 
> Hi,
> 
>    [{"bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>      "bt": 1320078429,
>      "ver": 3},
>     "bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>     "bt": 1320078429,
>     "ver": 3
>   },
>   {
>     "n": "temperature",
>     "u": "Cel",
>     "v": 27.2
>   },
>   {
>     "n": "humidity",
>     "u": "%RH",
>     "v": 80
>   }
> ]
>     "bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>     "bt": 1320078429,
>     "ver": 1
>   },
>   [
>     {
>       "n": "temperature",
>       "u": "Cel",
>       "v": 27.2
>     },
>     {
>       "n": "humidity",
>       "u": "%RH",
>       "v": 80
>     }
>   ]
> ]
> Or draft-01:
> 
> {
>   "bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>   "bt": 1320078429,
>   "ver": 1
>   "e": [
>     {
>       "n": "temperature",
>       "u": "Cel",
>       "v": 27.2
>     },
>     {
>       "n": "humidity",
>       "u": "%RH",
>       "v": 80
>     }
>   ],
> }
>> On Jan 13, 2016, at 5:57 AM, Cullen Jennings (fluffy) <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/ <https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/>
>> 
>> and 
>>

---

From: Christian Amsüss <[email-masked]>
Date: Fri, 15 Jan 2016 09:41:38 +0100

Hello Michael,

On Thu, Jan 14, 2016 at 06:43:38PM -0800, Michael Koster wrote:
> As pointed out, the streaming format can only be processed in
> streaming form. Storing this in memory using a JSON parser does not
> result in a structure that can be easily sorted or indexed by base
> element. If I were to design a parser for this, it would have a state
> machine that would consume the top level array elements in sequence
> and create an in-memory structure that looks like the structures
> described in draft-02:

thank you for clarifying what's meant by "forced to do stream
processing" (after all, I thought, you can still buffer things).

As I understand it, your concern is keeping your language's native
representation of the JSON structures as the internal state of your
SenML objects -- I'll argue that this is possible with -3 as well as
with -2.

> With these earlier structures I could use standard JSON parse tools
> and then selection algorithms that match the “bn” value and select the
> contents within that item’s scope (for patching, etc.)

I'll be assuming a Python implementation derived from
resource/SenmlHandler.py in your MachineHypermediaToolkit; other query
schemes should be similar to adapt. I'll also assume an older variant of
-03 where there are only two top-level elements allowed (the even-odd
scheme came from another thread) in the interest of brevity, but would
be happy to demonstrate a repeating-elements version if this is
unsatisfying.

Where a -01 implementation would be initialized as

    class Senml():
        def __init__(self, items=None, baseName=None):
            self._items = SenmlItems(items)
            self._senml = {}
            self._senml["e"] = self._items._items

replacing the last two lines with

            self._senml = [{}, self._items._items]

deals with the item locations, and base name assignment goes from

    self._senml["bn"] = baseName

to

    self._senml[0]["bn"] = baseName

. One or two more changes along similar lines should have the code base
covered, with no changes to SenmlItems being required.

That being said, I think that even with -01, such a data structure might
not be an ideal candidate for many applications, because it necessitates
(even though unimplemented in MachineHypermediaToolkit) that access to
both event names and values, they be added to the respective base
values for meaningful comparison. When elements are accessed repeatedly,
going over the whole structure at read time and building decompressed
values in memory would make things more straightforward as a whole. I
don't want to get into a discussion about implementation of memory
structures, though -- my point is that the mechanisms that incite such
discussion are present both in -01 and later.

> The other issue is that in draft-01 the element tag “e” maps to the
> XML serialization <e/> tags nicely. I have tools that process both XML
> and JSON that understand this.

Both -03 and -04 update their XML serializations to match the JSON
pretty. I see shortcomings in -03 (how would extensions with non-scalar
data store their elements in the head?), but they could probably be
mitigated by moving the head elements in a dedicated <b> element
and all the children into an <es> element.

> Also, this is easily extensible to add more element classes to the
> senml data model, which I have done of links and forms.

How is this not possible any more by adding them to the first
dictionary?

Extensions whose interpretations do not depend on "bn" or other entries
in the base dictionary are even processable easily by constrained
devices.

(I'm still unconvinced that a root "l":{links} is better than a
"v":{links} in elements, but would prefer to stick to the topic of "is
-03/-04 a regression" here).

I'm confident we can work a single SenML that works for all relevant
applications.

Christian

-- 
You don't become great by trying to be great. You become great by
wanting to do something, and then doing it so hard that you become great
in the process.
  -- Marie Curie (as quoted by Randall Munroe)

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 15 Jan 2016 10:17:29 +0100

Indeed, SenML -01 is optimized for implementations that ingest the
entire SenML document and then work on the result -- exactly what
happens in your Python implementation.

What -02 was trying to do was to also enable implementations operating
sequentially (what somewhat misleadingly has been called "streaming" --
well yes, enabling sequential processing is a prerequisite to enabling
streaming, but streaming is not the only reason for wanting to enable
sequential processing).

But of course this should not detract from the usefulness of the format
for ingest-bang implementations.
I think the structure in -02 is quite close to meeting both objectives.
I agree that the recent changes have led us away from that.

(I must admit I don't care as much about a complete analogy between XML
and CBOR formats; but as we now have found someone who cares about the
XML variant, we might be able to fix that.)

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Sat, 16 Jan 2016 14:27:51 +0100

Hello Cullen,

On Wed, Jan 13, 2016 at 01:57:02PM +0000, Cullen Jennings (fluffy) wrote:
> The objects can also have the base values right in the object and the
> base values apply to all the things in the object and any future
> objects in the array. 
>

The -03 draft is parsable with minimum RAM: I've assembled a crude
demo[1] that shows that given a list of known integer-valued resources
on a device, it is possible to parse incoming SenML-03 with less than 30
bytes of state, with support for arbitrary SenML extensions, all legal
JSON number representations and full URL concatenation semantics.

With the -04 draft, one would have to allocate a buffer for the "n"
value, because before the object finishes, the "bn" might change later
in the same object, and change the context in which it needs to be
interpreted. This could be remedied by making base and entry updates
mutually exclusive (arguably a less straight-forward design), but as
things are, I'd prefer -03.

Best regards
Christian

[1] https://gitlab.com/c.amsuess-energyharvesting/senml-03-streaming-parser-demo/tree/master

-- 
Christian Amsüss                      | Energy Harvesting Solutions GmbH
founder, system architect             | headquarter:
mailto:[email-masked]  | Arbeitergasse 15, A-4400 Steyr
tel:+43-664-97-90-6-39                | http://www.energyharvesting.at/
                                      | ATU68476614

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 07:24:41 -0800

Hi Christian,

1.
Thanks for the code example! It seems well optimized indeed.

It also serves to highlight one of the points. 

It’s more than just a matter of me liking my own favorite language (actually library) internal representation of JSON. Most modern libraries produce dictionary and array representations for parsed JSON that work more or less the same way.

The optimizations you recommend require a new string input parser, where I am using the JSON parser that comes with the library. My point was that I will need to either make my own string parser for SenML or make a re-parser with the state machine I described in order to use -02 + but there is another issue anyway:

One of the big advantages of using JSON is connecting the embedded world to the web world. To do this in a low-friction way, it’s good to be able to use well known tools and patterns. Streaming JSON 
2. 
Thanks also for looking into my code. I could easily use the -02 or -03 in the senml processing as you show. What is missing still from everything after -01 is the element tag that I am re-using to indicate links and forms in the document:

{
  "bn":"/3300/1/",
  "e":[
    {"n":"5700","v":"31.3"},
    {"n":"5602","v":"37.1"},
    {"n":"5601","v":"18.3"},
    {"n":"5605"},
    {"n":"5603","v":"0"},
    {"n":"5604","v":"100"},
    {"n":"5750","sv":"Inboard Bearing"}
  ],
  "l":[
    {"href":"","rel":"self","rt":"temperature","u":"C"},
    {"href":"5700","rt":"currentValue","u":"C"},
    {"href":"5602","rt":"maxValue","u":"C"},
    {"href":"5601","rt":"minValue","u":"C"},
    {"href":"5605","rt":"resetMaxMin"},
    {"href":"5603","rt":"minScale","u":"C"},
    {"href":"5604","rt":"maxScale","u":"C"},
    {"href":"5750","rt":"appType"}
  ]
}

I will think about other designs that do not depend on entity tags in the representation; maybe there is a better design for the content format in general. But I would still like to use SenML. Maybe not extend it but embed it in another new format…

In the meantime, I don’t have a good alternative yet.

Best regards,

Michael

PS Can we succinctly describe the use case for ordered serial processing of senml+json elements being important? From reading the draft, it seems like the important bit is saving the buffer memory for large responses. Is it a bigger deal with CBOR encoding or less important. 

Is the most important consideration being able to know the base values ahead of the data? 

Also, where is -03? there doesn’t seem to be a version -03 that you all are talking about on the IETF website:
https://tools.ietf.org/html/draft-jennings-core-senml-04 <https://tools.ietf.org/html/draft-jennings-core-senml-04>

> On Jan 16, 2016, at 5:27 AM, Christian Amsüss <[email-masked]> wrote:
> 
> Hello Cullen,
>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 16 Jan 2016 17:09:11 +0100

Michael Koster wrote:
> What is missing still from everything after -01 is the element tag that
> I am re-using to indicate links and forms in the document:

What would be wrong with including this in the metadata part of -02:

   [{"bn": "urn:dev:mac:0024befffe804ff1/",
     "l": [
      {"href"...}...
     ]},
    [ { "n": "voltage", "t": 0, "u": "V", "v": 120.1 },
      { "n": "current", "t": 0, "u": "A", "v": 1.2 } ]
   ]

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 09:43:55 -0800

Hi Carsten,

Yes, of course! This approach reminds me of JSON streaming, where actual delimiters are inserted into a text stream, but this is much better because it uses the JSON array itself to encapsulate elements that can be individually parsed.

That makes the outer element an array and the inner elements either array type or map map type, and the map can be extended to embed element types representing the links, forms, and data within the base element dictionary.

It would be useful if the order of top level element types could be arbitrary e.g.: [ {},[],[],[],{},[],{},{} ... ]

So for example, to use this format I will extend the base element map as you indicated:

[ 
  { "bn": "urn:dev:mac:0024befffe804ff1/",
    "l": [
       {"href"...}...
    ]

  },

If the order of top element types is optional, then the presence also could be, so I can adapt the new format by extending the base element to include links, forms, and items (getting rid of the “e” tag):

[ 
  { 
    "bn": “/light/brightness",
    "l": [
      { “href”: “change”,
        “rt”: [“change”,”form”],
        “ct”: “application/forms+senml+json”
      },
      { “href”: “tbr”,
        “rt”: [“property”, “item”, “param"],
        “ct”: “application/senml+json”
      },      
      { “href”: “tt”,
        “rt”: [“property”, “item”, “param"],
        “ct”: “application/senml+json”
      },
      { “href”: “actuations”,
        “rt”: “actuation”,
        “ct”: “application/collection+senml+json”
      }     
    ],
    “f”: [
      { “id”: “change”,
        "href”: “actuations”,
        “ct”: “application/collection+json”,
        “method”: “post”,
        “type”: “change”
        “template”: {
          “tbr”: “$targetbrightness”
          “tt”: “$transitiontime”
        }
      }
    ],
    “i”: [
      { “n”: “tbr”,
        “v”: “44.3”,
      },
      { “n”: “tt”,
        “v”: “1.0”,
      }
    ]	
  }
]

Having a top level array will allow me to accommodate collective operations on multiple object (bases) and allow the underlying implementation to chunk things at lease to the granularity of base objects mapped to top level elements.

So I’m OK with this if it allows me to use base elements as above.

Best regards,

Michael

> On Jan 16, 2016, at 8:09 AM, Carsten Bormann <[email-masked]> wrote:
> 
> Michael Koster wrote:

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 09:53:04 -0800

Here is a concrete proposal. 

My goal with the collection+senml content format is to create a format for machine hypermedia representations that is roughly analogous to a HTML web page in the web of documents world. 

The idea was a compact representation that can represent machine hypermedia data, links, and forms and would leverage existing formats and processors; link-format, senml, json. 

When I started, it looked like SenML was a good base due to what appeared to be an extension mechanism: the element tag that was an artifact of the XML serialization. I didn’t expect it to be removed because I have XML/JSON tools that programmatically generate that style as a lossless XML-JSON transform. This is generally useful because EXI is still a very popular format and in many cases more compact than CBOR.

If we can allow the base element to retain this functionality, and put the streaming support in the sequence of top level array elements, it seems like exactly what is needed. It retains backward compatibility with current senml implementations and allows some extensibility at both levels, while supporting sequential processing.

I propose to restore the base senml functionality of draft -01 within the base element definition in the new draft, that is to allow measurement data within the base element using the “e” key. To facilitate sequential processing, a topmost structure of array will allow an arbitrary sequence of base element maps and data element arrays as defined in draft -04.

Are there any issues with this construction? 

Best regards,

Michael

> On Jan 16, 2016, at 7:24 AM, Michael Koster <[email-masked]> wrote:
> 
> Hi Christian,
> 
> 1.
>> On Jan 16, 2016, at 5:27 AM, Christian Amsüss <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hello Cullen,
>> 
>> [1] https://gitlab.com/c.amsuess-energyharvesting/senml-03-streaming-parser-demo/tree/master <https://gitlab.com/c.amsuess-energyharvesting/senml-03-streaming-parser-demo/tree/master>
>> 
>> -- 
>> mailto:[email-masked] <mailto:[email-masked]>  | Arbeitergasse 15, A-4400 Steyr
>> tel:+43-664-97-90-6-39                | http://www.energyharvesting.at/ <http://www.energyharvesting.at/>
>>                                      | ATU68476614
>> [email-masked] <mailto:[email-masked]>

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 10:33:00 -0800

A small correction. draft -02 and -04 have different top level structures. -02 is [ {b} [ {e} {e} ] ]  and -04 is [ {b} {e} {e} ] Either one both will work for me, since the I will be able to extend the base item as needed. 

Michael

PS for correctness my example should look more like this:
[ 
  { 
    "bn": "/light/brightness",
    "l": [
      { "href": "",
        "rel": ["self"],
        "rt": ["brightness", "action"]
      },
      { "href": "change",
        "rel": ["form", "item"],
        "rt": ["change", "form"],
        "ct": "application/forms+senml+json"
      },
      { "href": "tbr",
        "rel": ["item"],
        "rt": ["property", "param"],
        "ct": "application/senml+json"
      },      
      { "href": "tt",
        "rel": ["item"],
        "rt": ["property", "param"],
        "ct": "application/senml+json"
      },
      { "href": "actuations",
        "rel": ["sub"],
        "rt": "actuation",
        "ct": "application/collection+senml+json"
      }     
    ],
    "f": [
      { "id": "change",
        “anchor": "../brightness"
        "rel": "change"
        "method": "post",
        "href": "actuations",
        "ct": "application/collection+json",
        "template": {
          "tbr": "$targetbrightness"
          "tt": "$transitiontime"
        }
      }
    ],
    "e": [
      { "n": "tbr",
        "v": "44.3",
      },
      { "n": "tt",
        "v": "1.0",
      }
    ]	
  }
]

> On Jan 16, 2016, at 9:53 AM, Michael Koster <[email-masked]> wrote:
> 
>> On Jan 16, 2016, at 7:24 AM, Michael Koster <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Christian,
>> 
>> 1.
>>> https://www.ietf.org/mailman/listinfo/core <https://www.ietf.org/mailman/listinfo/core>
>> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 16 Jan 2016 19:54:53 +0100

Michael Koster wrote:
> allow measurement data within the base element using the “e” key.

The reason -02 doesn't do that is that a sequentially operating
implementation may get process the "e" key before the base values,
possibly requiring it to buffer the values.  That's why we tried to
separate the measurements from the base values into different maps.

Grüße, Carsten

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 11:02:54 -0800

Yes, so if you want to stream, create an array with a base record and value records.

If you want to map, put your values into the base element with a “e” tag.

Michael

> On Jan 16, 2016, at 10:54 AM, Carsten Bormann <[email-masked]> wrote:
> 
> Michael Koster wrote:

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 11:03:57 -0800

The processing rule is each array element may (must in some cases) be processed in sequence

---

From: Michael Koster <[email-masked]>
Date: Sat, 16 Jan 2016 11:05:56 -0800

And maps must be consumed before processing. So keep your maps small and use arrays

> On Jan 16, 2016, at 11:03 AM, Michael Koster <[email-masked]> wrote:
>

---

From: Christian Amsüss <[email-masked]>
Date: Mon, 18 Jan 2016 12:05:00 +0100

Hello Michael,

it appears that there are three concepts both being called or related to
"streaming" that get mixed up in this thread, I'll try to flesh them out
fist, describing their purpose, status of drafts, and interaction with
client and server side.

A. The draft paragraph about "store or transmit SenML in a stream-like
  fashion" in the "Multiple Datapoints" section:

  This is more about transportation, and implies that there is some kind
  of boundary between elements at which transmission can pause.
  
  Such boundaries are present in all drafts so far.

  It appears to me that this would primarily work with HTTP in a fashion
  similar to long polling, which is something both communicating parties
  need to be aware of (all drafts: "MUST specify that they are doing
  this"), lest they time out the connection.

B. Buffer-less operation:

  For very small applications (eg. using uIP and few kB of RAM), it is
  desirable to use data formats that never require back-seeking, that
  limit back-seeks to a fixed length or that can do with a fixed-length
  buffer to hold information for that. The specification does not need
  to actively describe those operations, requiring some basic structure
  is sufficient.

  It has been suggested earlier on this list that -01 with an additional
  requirement that "bn" and "bt" to come first in the dictionaries would
  allow this mode of operation too -- truth, but hard to achieve with
  generic JSON implementations and semantics.

  Thus, this goal is only achieved by drafts -02 and later. (This is
  also what my demo is about, which shows that -03 is particularly
  suitable).

  That this only an issue for the receiving side (the sender is free
  to choose element sequences as is most practical for it anyway, as
  long as it follows the specification). Having this negotiable would
  defeat its purpose: If receivers may rely on it, we can't allow
  senders to not support it.

C. Multiple base elements:

  The idea to have multiple base elements came up independently of the
  others, and gives smaller data streams for cases in which different
  sensors' time series are produced, or when there are larger gaps in a
  timeline. The proposal worked well with the array-as-root scheme
  envisioned with exactly 2 elements for B., and was implemented
  together.

  It is present in -03 and -04.

  Senders are free not to use it. In my impression, this is easily to
  implement in receivers. (Even if they desire to directly access
  in-memory representations of JSON; the key to an entry might then be
  (list-number, list-position) instead of just list-position).

> PS Can we succinctly describe the use case for ordered serial
> processing of senml+json elements being important? From reading the
> draft, it seems like the important bit is saving the buffer memory for
> large responses. 

The one I'm arguing here is B., the buffer-less operation. The use case
are devices with limited heap memory (say 4k for incoming and outgoing
packages). I'm using SenML in core-interfaces batches[1] to get and set
device state. When those devices announce supporting batch operation,
they need to accept SenML PUTs to as many resources as are batched
together, so far without arbitrary constraints. Due to the proposed
extensibility of SenML, even if I limited the number of resources in a
batch, I couldn't predict the maximum size of an incoming update.

Now when the update arrives and a SenML that does not satisfy B. comes
in, all I can do is to store the whole message body and parse it when
the last block arrived. If the sender updated too many resources or used
too large extensions, I'd need to 4.13-Request-Entity-too-large out.
Still, until then (particularly until the overflowing package arrives),
memory is clogged with unparsable data.

> Is it a bigger deal with CBOR encoding or less important. 

Whether CBOR or JSON is being parsed is almost irrelevant here. The only
difference that comes to my mind is that with CBOR, we might be able to
prescribe a serialization where "bn" and "bt" always precede "e" even in
-01-style, but even that would make it harder to utilize for whomever
implement this with a native-object-style library.

> Is the most important consideration being able to know the base values

This is about all information that may be in the dictionary and modifies
how to interpret the elements' data. In core SenML, that information is
the base attributes, and the version -- that is, all the possible root
variables. (In a scenario .)

For extensions, this completely depends on them unless the receiver
chooses to ignore them anyway. As far as I understand, your link format
extension could add items to a core-interfaces linked batch, and then
set a value in one go; for that, the extension data would need to
precede the elements in parsing. Other extensions (say, metadata about
when to expect changes) may be useful no matter where in the data stream
it appears.

[1] https://tools.ietf.org/html/draft-ietf-core-interfaces-04#section-6.2

> https://tools.ietf.org/html/draft-jennings-core-senml-04 <https://tools.ietf.org/html/draft-jennings-core-senml-04>

The data tracker (or rather tools.ietf.org) seems not to take drafts
submitted on the same day well. You can view it as [2].

[2] https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/?include_text=1

On Sat, Jan 16, 2016 at 07:24:41AM -0800, Michael Koster wrote:
> The optimizations you recommend require a new string input parser,
> where I am using the JSON parser that comes with the library. My point
> was that I will need to either make my own string parser for SenML or
> make a re-parser with the state machine I described in order to use
> -02 +[...]

I don't see yet where the state machine comes in. Surely, much
state-machingin is going on in the optimized example, but when it comes
to the core difference from -01 to -02, that is, that

    {any-key-but-e: any-data, "e": elements-list}

becomes

    [{any-key-but-e: any-data}, elements-list]

, I don't see how this is not changing access from _senml[key] to
_senml[0][key] or _senml["e"] to _senml[1].

(If it's about the C. multiple-base-elements, I'd ask to keep these
issues separate -- if C. breaks too much, although I like it, we could
have a -02 with only two elements and still satisfy B., but I don't
think we need to resort to that.)

> [...] but there is another issue anyway:
> 
> One of the big advantages of using JSON is connecting the embedded
> world to the web world. To do this in a low-friction way, it’s good to
> be able to use well known tools and patterns.

I fully agree that easy use of established web tools is essential here:
but these very threads are where we should come up with mechanisms that
work both with them and constrained devices.

There is one point where I think that state-machining over incoming JSON
objects does make sense, that is when it comes to names -- I like to see
them as URIs (that's consistent with the examples given, but not fully
required in the drafts), and when the "n" elements are relative URIs,
they are not useful until joined with the base name. But that is not a
change introduced in -01, but a preference of mine that is reflected in
the demo to show that relative URIs are not that complicated even for
embedded systems. (Existing web tools usually have their ways of joining
URIs shipped).

> Thanks also for looking into my code. I could easily use the -02 or
> -03 in the senml processing as you show. What is missing still from
> everything after -01 is the element tag that I am re-using to indicate
> links and forms in the document:

I did see that you had extensions in your code, but as Carsten
suggested, the "l" could stay in the first dictionary alongside the "bn"
and "ver". Maybe the name "base dictionary" is suboptimal here -- the
dictionary itself is not something that gets somehow prefixed to the
following entries, it's just the place where the base attributes reside
along with other things, like "ver" and extensions.

> It would be useful if the order of top level element types could be arbitrary e.g.: [ {},[],[],[],{},[],{},{} ... ]

That's something I could well see in a -05. It would also make minimal
SenML files that don't use any base attributes a little smaller, and
embedded parsing wouldn't suffer from it AFAICT.

On Sat, Jan 16, 2016 at 10:33:00AM -0800, Michael Koster wrote:
>     "l": [ ... ],
>     "f": [ ...],
>       { "n": "tbr", "v": "44.3", },
>       { "n": "tt", "v": "1.0", }
>     ]
>   }
> ]

Just let's please not allow the "e" element any more. Anything with a
top-level array is incompatible with older draft users anyway.

Best regards
Christian

---

From: Michael Koster <[email-masked]>
Date: Mon, 18 Jan 2016 07:42:10 -0800

Hi Christian,

Thanks for the comprehensive reply. I understand better now the bigger picture.

I have added a few comments below. 

> On Jan 18, 2016, at 3:05 AM, Christian Amsüss <[email-masked]> wrote:
> 
> Hello Michael,
> 
Yes, I understand. I’m assuming that any array structure with small enough elements
will satisfy this goal. In this case the base needs to be remembered at the receiver 
anyway, so  having base in a separate element makes sense. We are receiving data 
with a state machine which stores each base value as a new state and context from 
which to interpret new data elements.

>  generic JSON implementations and semantics.
Limiting the size of each element in the array will limit the back-seeking needed
thus the size of the buffer will need to be related to the maximum size of an array
element. Having the outermost structure be an array enables that in the drafts
post -01.

>  senders to not support it.
This to me is key. For reliable operation the system designer needs to insure 
about the maximum message sizes and buffers. The limit on the size of any 
array element, base or data, can’t be specified in this RFC, right?

>  (list-number, list-position) instead of just list-position).
I am using a multiple base format in my reference implementation.
Some of my examples are for just one element in the outermost array.
> memory is clogged with unparsable data.
My use case is the hypermedia collection in CoRE Interfaces-04, which has batch 
and linked batch (and group). I like the idea of being able to do these multi-resource 
updates in chunked operations on small devices. It seems the array format is suitable
for this use case. It does require a state machine at the receiver to map all of the elements
to the most recently sent base, but that is the nature of buffer-less processing.

> [1] https://tools.ietf.org/html/draft-ietf-core-interfaces-04#section-6.2 <https://tools.ietf.org/html/draft-ietf-core-interfaces-04#section-6.2>

The base and elements need to both be known to start the action. There is not 
a requirement  to know the base any time in advance of the elements. This is 
why mapping is OK, it just requires a buffer.
> think we need to resort to that.)

The state machine comes in from assuming multiple base format. If there
is exactly one base and zero or more data elements, the array semantics 
can be used as you show.

If there is an array of maps, each with base and elements, then it’s natural 
to sequentially process them as separate base items.

If it’s an array with a base map followed by data elements, I will probably 
transform that to a map anyway and coalesce the data elements rather 
than refer to elements by [0] and [1:] in my higher level code.

senml[“e”] has semantic meaning, but I wrap it an an elements class anyway.
I guess I was using the format itself as a data model, thus refer to objects as 
“senml” objects. That is what feels broken by the new formats. I should not
tie the data model to the format.

> embedded systems. (Existing web tools usually have their ways of joining
It is very important to the proper operation of collections and link embedding 
that the base name (uri of the collection) and resource names in the collection
(relative references from that base) are consistently and easily handled
(by the tools for the developer). 

Collection templates need to be reusable, therefore relative references to 
late binding base URIs and hypermedia controls are important.

That is our motivation to use the senml data model in the CoRE Interfaces 
collection models. It also why to me it makes sense to have a format where 
links and items are combined, for hypermedia operation.
> along with other things, like "ver" and extensions.
Yes, putting the links in with the base makes perfect sense. this would 
allow hypermedia based assembly of the following data elements. The
links would have “href” items that can be used to match and select 
items by name (“n”:) in the data stream.
> embedded parsing wouldn't suffer from it AFAICT.
It would give the system designer more options to optimize the semantics 
while keeping the required buffer size small.

> top-level array is incompatible with older draft users anyway.
We seem to be in agreement mostly. If it is important to get rid of the “e”
tag and it’s remnant from the flattened <e/> in the xml, that makes sense.
I am not attached at all to using the “e” tag.

Best regards,

Michael

---

From: Michael Koster <[email-masked]>
Date: Mon, 18 Jan 2016 08:23:00 -0800

Some more thoughts on data model.

It turns out to be more than just convenient that the senml representation 
itself is an explicit mapping of a data model.

the “e” tag is more than just a semantic glitch in converting XML to JSON. 
To preserve the semantics of the data model, the data items as a set are 
related to the base at the same scope level in the model. The XML form
is “flattened” and JSON is not. 

I used this to extend the data model by adding a links class at the same 
scope level as base name and items.

This is how the collection with links and items can be expressed in both
machine and human comprehensible form as a model:

{
"bn":"/3300/1/",
“i":[
{"n":"5700","v":"31.3"},
{"n":"5602","v":"37.1"},
{"n":"5601","v":"18.3"},
{"n":"5605"},
{"n":"5603","v":"0"},
{"n":"5604","v":"100"},

Base name, links, and items all are easily mapped to property names
using a simple JSON-LD context and can be understood and modeled 
by higher level tools as-is. Removing the “i” or “l” tags gives the modeling 
tool nothing to use to identify property instances (like base name, links, 
and items). This is extensible by adding new property  classes (element 
tags).

It is very useful to have the representation match the model so closely.
I think there will be future generalizations of this pattern that don’t need
such explicit representation formats, but for now I find it more than just
convenient.

If the representation wasn’t structured in this way, I would need to 
create an internal representation that is, in order to make use of
the model tools that I’m already using. Doing this would IMO not
be good for the human understanding, which is important in the
phase here where people are learning.

Maybe there is a way to preserve the model semantics in the 
representation and at the same time support efficient sequential 
processing of elements.

Best regards,

Michael

---

From: Christian Amsüss <[email-masked]>
Date: Mon, 18 Jan 2016 17:26:21 +0100

Just a quick reply before going through the whole mail:

> element. Having the outermost structure be an array enables that in the drafts

With all the "e"-less proposals, for the applications I'm describing, no
element size limits (neither in serialized bytes length nor in elements)
are required at all as there is no back-seeking.

Best regards
Christian

---

From: Christian Amsüss <[email-masked]>
Date: Mon, 18 Jan 2016 17:58:11 +0100

Again a short reply before going through the full mail:

On Mon, Jan 18, 2016 at 08:23:00AM -0800, Michael Koster wrote:
> {
> "bn":"/3300/1/",
> “i":[ {"n":"5700","v":"31.3"} ],
> "l":[
>   {"href":"","rel":"self","rt":"temperature core.s","u":"C"},
>   {"href":"5700","rt":"currentValue","u":"C"}
> ]
> }
> 

I'm curious: Is JSON-LD at all able to derive, from the above (I took
the liberty to abbreviate and add a second rt), statements like

<http://.../3300/1/> a core:s, some:temperature ;
    some:self <http://.../3300/1/>; core:unit "C" ;
<http://.../3300/1/5700> a some:currentValue ;
    core:grp <http://.../3300/1>;
    some:value "31.3"

? That would require matching the bn and n for URI concatenation and
whitespace-splitting the rt attribute.

If it is possible -- great, and it might serve as a criterion for a
bufferlessly parsable SenML that it can be transformed using JSON-LD.
But as far as I remember, it can't do that even for -01, and unless that
can be fixed, I don't see much of a point in using it as an example.

Best regards
Christian

---

From: Michael Koster <[email-masked]>
Date: Mon, 18 Jan 2016 09:30:47 -0800

> On Jan 18, 2016, at 8:58 AM, Christian Amsüss <[email-masked]> wrote:
> 
JSON-LD needs JSON document format to start with, so I first transform 
link-format to link-format+json which makes multi-valued attributes an array. 

The tags are mapped to property names in the JSON-LD context using maps
like in this example context for my W3C WoT schema. 

"capabilities": {"@id": "ts:hasCapability", "@type": "@id"},
"actions": {"@id": "ts:hasAction", "@type": "@id"}
“events": {"@id": "ts:hasEvent", "@type": "@id"}
“properties": {"@id": "ts:hasProperty", "@type": "@id"}

@type:@id maps the key to a URI. for senml I might start with these 
maps in the context (where some is also defined somewhere).

“bn": {"@id": “some:baseURI", "@type": "@id"}
“i": {"@id": “some:collectionItems", "@type": "@id”}
“l": {"@id": “some:collectionLinks", "@type": "@id"}
“n": {"@id": “some:baseURI", "@type": "@id”}
“v": {"@id": “some:numberValue", "@type": "@id"}
“vs": {"@id": “some:stringValue", "@type": "@id"}
“vb": {"@id": “some:booleanValue", "@type": "@id”}

and define xsd types, etc. at the referenced URI in some namespace, 
but there are other ways that the tools could also comprehend.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Mon, 25 Jan 2016 22:14:44 +0000

> On Jan 13, 2016, at 7:30 AM, Carsten Bormann <[email-masked]> wrote:
> 

Seems like a good argument for only the first record can have base values. Any objections to making that change for next version ?

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Mon, 25 Jan 2016 22:15:54 +0000

This message really helps clarify things. Thanks. More inline

On Jan 18, 2016, at 4:05 AM, Christian Amsüss <[email-masked]<mailto:[email-masked]>> wrote:

Hello Michael,

agree and I think a different mine type can be used for both sides to understand they are doing this. In some cases the stream may never end and the JSON to close the data structures would never be sent so at some level it’s not even valid JSON.

I think supporting this is important. It’s good for small things but it also tends to support server implementations that are large but need to be able to process really a lot of data quickly.

I’m starting to think that the use case for multiple base elements is very weak and that perhaps we should only allow a single set of base units which are before any of the measurements. If we get this for free or cheap, great, but not something I care a lot about.

Is the most important consideration being able to know the base values
ahead of the data?

The data tracker (or rather tools.ietf.org<http://tools.ietf.org>) seems not to take drafts
submitted on the same day well. You can view it as [2].

becomes

It might be better if specification which used SenML said the combination of bn+n MUST be a URI and perhaps even constrain what type of URI are OK.

if any of the post -01 version broke the ability to add extensions, that is just an mistake or my part and we can fix it. Allowing JSON-LD style links in the bn makes sense to me.

I need to send more justification about this to the list but looking at common JSON parsers for languages that do no allow variant arrays, it seems that many of theses libraries don’t easily support JSON with array of different types.

For example the golang JSON stuff. You can see how this works at

https://github.com/cisco/senmlCat/blob/master/senmlCat.go

On a side note, senmlCat can covert SenML to and from various formats and can also act as a HTTP server that receives POST of SenML and then writes them files, or influxdb, or kafka. I would not call it more alpha than done but hey, send a pull request.  I could not talk about some of the SenML project but having some open source code to point at that we can talk about is useful.

If you poke thought the libraries at http://www.json.org/ for a language like C, you can see they either do mallocs for every element in the list - which gets very slow compared to something like protobuf - or the schema tells the system the type of element in the array.

Now clearly JSON, XML, CBOR etc can all support variant arrays. And clearly in any language you can write a parser and data structure that allows them (even Fortran). We are just talking about performance and convenience  of using variant arrays in a language like C. I think it would very nice to be able to avoid variant arrays if we don’t need them.

Best regards
Christian

---

From: Stephen Farrell <[email-masked]>
Date: Mon, 25 Jan 2016 22:43:32 +0000

I'm willing to take the serious risk of diving at entirely
the wrong place, (and apologies in advance if so;-), but...

[1] seems to be using a MAC address to identify a sensor
reading. Ick. Even if a sensor never moves, that can be
bad if the sensor reading is only emitted as a result of
some privacy-relevant occurrence in the environment. If
a sensor gets carried about then it can be much worse.

I really hope we can provide something better than
such a long-term stable and easily tracked identifier,
even if that requires a bit more work. I don't think
section 13 [2] is really there yet either.

Is there text in some other place that could be
referenced or is this the document that ought define
some way to use identifiers that are less privacy
unfriendly? I do think we (IETF) should somewhere do
the work to define some such identifiers so that the
folks developing sensors and sinks for sensor data
can do a lot better.

Cheers,
S.

[1] https://tools.ietf.org/html/draft-jennings-core-senml-04#section-6.1.2
[2] https://tools.ietf.org/html/draft-jennings-core-senml-04#section-13

---

From: Christian Amsüss <[email-masked]>
Date: Tue, 26 Jan 2016 08:35:16 +0100

On Mon, Jan 25, 2016 at 10:14:44PM +0000, Cullen Jennings (fluffy) wrote:
> Seems like a good argument for only the first record can have base
> values. Any objections to making that change for next version ?

Not from my side; I'd suggest going as far as forbidding per-item
members in the first record. (Extensions would be free to allow their
members in both a (or the) base object and in item objects).

Best regards
Christian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Jan 2016 10:26:56 +0100

That would be a radical simplification, indeed:

senml = [header, * measurement]

header = {
      ? bn: tstr,    ; Base Name
      ? bt: number,  ; Base Time
      ? bu: tstr,    ; Base Units
      ? ver: number, ; Version
      * tstr => any, ; (Extension)
}

measurement = {
      ? n: tstr,     ; Name
      ? u: tstr,     ; Units
      ? v: float,    ; Value
      ? sv: tstr,    ; String Value
      ? bv: bool,    ; Boolean Value
      ? s: float,    ; Value Sum
      ? t: number,   ; Time
      ? ut: number,  ; Update Time
}

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Tue, 26 Jan 2016 13:01:26 +0100

> “n": {"@id": “some:baseURI", "@type": "@id”}

I think that this is one point where any of the existing SenML drafts
fail to produce meaningful statements for arbitrary SenML documents;
with a context as sketched above, the bn would be ignored completely,
and the value of the event's n would be a concatenation of the document
URI and the n.

After processing this into RDF, there wouldn't even be enough
information left to post-process bn into the names even if that were
somehow more practical than pre-processing the JSON.

Unless we go full JSON-LD by changing "bn" into "@base" in some way, or
JSON-LD is extended to allow something like {"@base":{"@query":
"[0].bn"}} (like xpath for json, no idea if something exists or is
feasible), I don't see how we can have the semantics of SenML
transformed into RDF by JSON-LD.

We could probably describe a subset of SenML that works for a specific
context (for example by completely ruling out bn), but how useful would
that be out in the wild?

(Similar issues exist with the whitespace separated fields in the rt
field in link-format. draft-ietf-core-links-json-04 mentions JSON-LD,
but at the same time makes "[n]o attempt [...] to decode the possibly
space-separated values for rt=" -- as long as neither links-json nor
JSON-LD deals with string splitting, we can't have JSON-LD parsing for
generic link-format).

To compare this to RDFa which could probably work with the XML
serialization as JSON-LD could work with the JSON one: To have support
for the HTML base element, RDFa had to specify (X)HTML+RDFa as a "host
language" to cover the base tag -- which basically means that whatever
processes HTML+RDFa needs to do preprocessing.

Unless those issues can be managed in any way, I don't see much point in
engineering the SenML standard towards JSON-LD convertibility. How do
you overcome them in your own applications?

Best regards
Christian

---

From: Michael Koster <[email-masked]>
Date: Tue, 26 Jan 2016 20:28:31 +0700

Hi Christian,

I'm not sure how you expect the document conversion to work, but I used a JSON-LD context to transform a SenML-draft-01 format file to semantic triples in Nquads using the processor at http://json-ld.org/playground/ <http://json-ld.org/playground/> . The JSON-LD is simply the SenML file with the @context statement inserted after the opening bracket.

The SenML is a representation of an IPSO Smart Object:

{
    "@context" : "http://thingschema.org",
    "bn":"/3303/1/",
    "i":[
    {"n":"5700","v":"31.3","u":"Cel"},
    {"n":"5701","sv":"Cel"},
    {"n":"5602","v":"87.1","u":"Cel"},
    {"n":"5601","v":"18.3","u":"Cel"},
    {"n":"5605"},
    {"n":"5603","v":"0","u":"Cel"},
    {"n":"5604","v":"260","u":"Cel"},
    {"n":"5750","sv":"Inboard Bearing"}
    ],
    "l":[
    {"href":"","rel":"self","rt":"temperature","u":"Cel"},
    {"href":"5700","rt":"currentValue","u":"Cel"},
    {"href":"5701","rt":"units"},
    {"href":"5602","rt":"maxValue","u":"Cel"},
    {"href":"5601","rt":"minValue","u":"Cel"},
    {"href":"5605","rt":"resetMaxMin"},
    {"href":"5603","rt":"minRange","u":"Cel"},
    {"href":"5604","rt":"maxRange","u":"Cel"},
    {"href":"5750","rt":"appType"}
    ]
}

And here are semantic triples generated from the SenML, having used http://json-ld.org/playground/ <http://json-ld.org/playground/> to process the elements into RDF subjects, properties, and values and serialize into N-quads.

_:b0 <http://thingschema.org/schema#baseName> "/3303/1/" .
_:b0 <http://thingschema.org/schema#hasItem> _:b1 .
_:b0 <http://thingschema.org/schema#hasItem> _:b2 .
_:b0 <http://thingschema.org/schema#hasItem> _:b3 .
_:b0 <http://thingschema.org/schema#hasItem> _:b4 .
_:b0 <http://thingschema.org/schema#hasItem> _:b5 .
_:b0 <http://thingschema.org/schema#hasItem> _:b6 .
_:b0 <http://thingschema.org/schema#hasItem> _:b7 .
_:b0 <http://thingschema.org/schema#hasItem> _:b8 .
_:b0 <http://thingschema.org/schema#hasLink> _:b10 .
_:b0 <http://thingschema.org/schema#hasLink> _:b11 .
_:b0 <http://thingschema.org/schema#hasLink> _:b12 .
_:b0 <http://thingschema.org/schema#hasLink> _:b13 .
_:b0 <http://thingschema.org/schema#hasLink> _:b14 .
_:b0 <http://thingschema.org/schema#hasLink> _:b15 .
_:b0 <http://thingschema.org/schema#hasLink> _:b16 .
_:b0 <http://thingschema.org/schema#hasLink> _:b17 .
_:b0 <http://thingschema.org/schema#hasLink> _:b9 .

_:b1 <http://thingschema.org/schema#resourceName> "5700" .
_:b1 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .
_:b1 <http://thingschema.org/schema#value> "31.3" .

_:b10 <http://thingschema.org/schema#href> "5700" .
_:b10 <http://thingschema.org/schema#resourceType> "currentValue" .
_:b10 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

_:b11 <http://thingschema.org/schema#href> "5701" .
_:b11 <http://thingschema.org/schema#resourceType> "units" .

_:b12 <http://thingschema.org/schema#href> "5602" .
_:b12 <http://thingschema.org/schema#resourceType> "maxValue" .
_:b12 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

_:b13 <http://thingschema.org/schema#href> "5601" .
_:b13 <http://thingschema.org/schema#resourceType> "minValue" .
_:b13 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

_:b14 <http://thingschema.org/schema#href> "5605" .
_:b14 <http://thingschema.org/schema#resourceType> "resetMaxMin" .

_:b15 <http://thingschema.org/schema#href> "5603" .
_:b15 <http://thingschema.org/schema#resourceType> "minRange" .
_:b15 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

_:b16 <http://thingschema.org/schema#href> "5604" .
_:b16 <http://thingschema.org/schema#resourceType> "maxRange" .
_:b16 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

_:b17 <http://thingschema.org/schema#href> "5750" .
_:b17 <http://thingschema.org/schema#resourceType> "appType" .

_:b2 <http://thingschema.org/schema#resourceName> "5701" .
_:b2 <http://thingschema.org/sv> "Cel" .

_:b3 <http://thingschema.org/schema#resourceName> "5602" .
_:b3 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .
_:b3 <http://thingschema.org/schema#value> "87.1" .

_:b4 <http://thingschema.org/schema#resourceName> "5601" .
_:b4 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .
_:b4 <http://thingschema.org/schema#value> "18.3" .

_:b5 <http://thingschema.org/schema#resourceName> "5605" .
_:b6 <http://thingschema.org/schema#resourceName> "5603" .
_:b6 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .
_:b6 <http://thingschema.org/schema#value> "0" .

_:b7 <http://thingschema.org/schema#resourceName> "5604" .
_:b7 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .
_:b7 <http://thingschema.org/schema#value> "260" .

_:b8 <http://thingschema.org/schema#resourceName> "5750" .
_:b8 <http://thingschema.org/sv> "Inboard Bearing" .

_:b9 <http://thingschema.org/rel> "self" .
_:b9 <http://thingschema.org/schema#href> "" .
_:b9 <http://thingschema.org/schema#resourceType> "temperature" .
_:b9 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

I think this result is quite useful to be able to use the information in the context of fully qualified references and nodes of triples that describe the original SenML.

Best regards,

Michael

> On Jan 26, 2016, at 7:01 PM, Christian Amsüss <[email-masked]> wrote:
>

---

From: Christian Amsüss <[email-masked]>
Date: Tue, 26 Jan 2016 17:40:36 +0100

Hi Michael,

On Tue, Jan 26, 2016 at 08:28:31PM +0700, Michael Koster wrote:
>     {"n":"5700","v":"31.3","u":"Cel"}, ...
>     ],
>     "l":[

thanks for the example, that's a good starting point to work with.

> And here are semantic triples generated from the SenML, having used
> http://json-ld.org/playground/ <http://json-ld.org/playground/> to
> process the elements into RDF subjects, properties, and values and
> serialize into N-quads.
> 

These are quads that contain the information of the SenML, but not
really in an RDFish way.

RDF is all about using URIs to describe things. What I'd expect of such
a conversion when done right would be N-triples like

_:b1 <http://thingschema.org/schema#about> </3303/1/5700> .
_:b1 <http://thingschema.org/schema#value> "31.3"^^xsd:float .
_:b1 <http://thingschema.org/schema#unitsOfMeasure> "Cel" .
</3303/1/5700> <http://thingschema.org/schema#resourceType> "currentValue" .
</3303/1/5700> <http://thingschema.org/schema#unitsOfMeasure> "Cel" .

where statements are made about actual resources. _b:1 would still be a
blank node, because it reflects a single description of the resource's
state (were it a time series, the necessity for that would be more
obvious). A converter might want to even emit a statement like

_:b1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://thingschema.org/schema#Item> .

to make that clear.

The output produced by the thingschema.org still reads a little like a
combination of text fragments; to compare it with relational databases,
it looks to me as if there were no foreign keys in use yet, and
everything were stored in text fields. The output is RDF, but it can't
effectively be used as RDF yet.

> I think this result is quite useful to be able to use the information
> in the context of fully qualified references and nodes of triples that
> describe the original SenML.

I do not understand what you mean by this, could you elaborate? AFAICT,
were I fed the N-quad output from above, I couldn't even establish any
relationship between the "/3303/1/" URI fragment and the "5700", let
alone use that in a query about </3303/1/5700>.

There are things brewing at W3C's Web of Things initiative[1], there has
for some time been work on semantically connected IoT devices[2], and I
am confident that RDF metadata are the way to go.

But both with a SenML that looks anything like the drafts so far and
with draft-ietf-core-links-json, I think that JSON-LD does not have what
it takes to produce the meaning of the respective JSON representations
for the semantic web.

I've long ago asked on this list about a canonical RDF representation of
a link-format resource. Back then, there was nothing, and I had only a
rough idea of what I'd need as well. I'd be all in for going ahead here
and sketching up what a generic link-format document would look like in
an RDF representation, and possibly SenML as well; that should be a
separate discussion, though, and not deeply interfere with the SenML
process itself. I could sketch up some parts that've been brewing with
me and are partially implemented as well -- would that be suitable for
this list?

Best regards
Christian

[1] http://www.w3.org/WoT/
[2] https://iotdb.org/

---

From: Michael Koster <[email-masked]>
Date: Wed, 27 Jan 2016 03:21:58 +0700

> On Jan 26, 2016, at 11:40 PM, Christian Amsüss <[email-masked]> wrote:
> 
> Hi Michael,
> 
Hmm so I can't know from the schema that baseName and href are concatenated to form /3303/1/5700?
That seems to be all that's missing.

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 27 Jan 2016 08:05:19 +0100

> Hmm so I can't know from the schema that baseName and href are concatenated to form /3303/1/5700?

For all I know, no, neither in the JSON-LD step (because "@base" can be
only a single string and not some JSON-path), nor at the RDF step (if
there were a trick to synthesize the statements, I don't think that most
tools could use them).

> That seems to be all that's missing.

That, and string splitting (when it comes to parsing the "l" entries'
"rt" attributes).

They are annoyingly small details, but unless JSON-LD can be extended to
be more XSLT-like, I think they are show stoppers here.

Best regards
Christian

---

From: Michael Koster <[email-masked]>
Date: Wed, 27 Jan 2016 14:32:15 +0700

> On Jan 27, 2016, at 2:05 PM, Christian Amsüss <[email-masked]> wrote:
> 
I don't see where any string splitting is needed. The "l" entries are all JSON parseable, as are the rt attribute values.

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 27 Jan 2016 09:19:41 +0100

> I don't see where any string splitting is needed. The "l" entries are all JSON parseable, as are the rt attribute values.

The rt and if attributes of application/link-format requires that all
resource types assigned to a resource be stored in a single rt=""/if=""
attribute in a space separated fashion (rfc6690 3.1, as is also the case
for rel per rfc5988), and draft-ietf-core-links-json-04 makes no attempt
to split those up (2.2).

If resource types were to be used in a way they can be queried in an RDF
context, the statements produced from <x>;if="core.s some.delayable"
should at least contain[1]

<x> <http://thingschema.org/schema#if> <http://thingschema.org/if#core.s> .
<x> <http://thingschema.org/schema#if> <http://thingschema.org/if#some.delayable> .

because a statement like

<x> <http://thingschema.org/schema#if> "core.s some.delayable" .

would again mean shifting the load of string parsing into the RDF
domain, where the expectation is that reasoning can be done based on URI
metadata and not on string parsing. (Again, the analogy of the
relational database comes into my mind: You wouldn't want to have
resource types you'll query for in a space-separated text field of a
database).

Also, again drawing the parallel to RDFa, when this HTML document
test.html is parsed as RDFa (eg. using the rapper tool that is aware of
the registered describedby relationship)

<html prefix="demo: http://example.com/demo"><link rel="demo:next describedby unspecified" href="./next-address" /></html>

then the resulting statements are

<./test.html> <./unspecified> <./next-address> .
<./test.html> <http://example.com/demo#next <./next-address> .
<./test.html> <http://www.w3.org/2007/05/powder-s#describedby> <./next-address> .

Best regards
Christian

[1] If it turns out that JSON-LD is unsuitable for this task anyway, as
I have the impression which I'm trying to convey, then I'd even propose
introducing URI CURIEs into link-format, so we have strong meaning for
some.temperature:

<http://example.org/some/v1#>,rel="ns",curie="some";
<x>;if="core.s some.delayable";rt="some.temperature"

would be interpreted (for a predefined meaning of core) as

<x> <http://thingschema.org/schema#if> <http://thingschema.org/core#s> .
<x> <http://thingschema.org/schema#if> <http://example.org/some/v1#delayable> .

but details of that are for if and when it turns out that I am right
about JSON-LD's applicability.

---

From: Michael Koster <[email-masked]>
Date: Wed, 27 Jan 2016 18:24:10 +0700

> On Jan 27, 2016, at 3:19 PM, Christian Amsüss <[email-masked]> wrote:
> 

Oh, I see. I think this is an omission in the document. As I understand, it is the 
intention for the JSON serialization of links with multiple attribute values to use 
JSON arrays to represent those values.

I will discuss with the draft editor and confirm this.

My examples are built from JSON serializations that use array type to represent 
multiple attribute values.

---

From: Michael Koster <[email-masked]>
Date: Thu, 28 Jan 2016 01:54:00 +0700

Excuse me, I was wrong. On reading the draft https://tools.ietf.org/html/draft-ietf-core-links-json-04 <https://tools.ietf.org/html/draft-ietf-core-links-json-04>
there is clear normative text in the draft that requires multiple valued parameters to be represented as 
JSON arays (sec 2.2):

   If a Link attribute ("parmname") is present more than once in a
   "link-value", its values are then represented as a JSON array of JSON
   string values; this array becomes the value of the JSON name/value
   pair where the attribute name is the JSON name.  Attributes occurring
   just once MUST NOT be represented as JSON arrays but MUST be directly
   represented as JSON strings. 

There is no example of this in the draft, nor is there a formal syntax production to show it, but the text is clear.

Sorry for the confusion.

Michael

> On Jan 27, 2016, at 6:24 PM, Michael Koster <[email-masked]> wrote:
> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Jan 2016 20:14:26 +0100

Michael Koster wrote:
> There is no example of this in the draft, nor is there a formal syntax
> production to show it, but the text is clear.

A link relation can be present more than once, and this rule defines how
to handle this.
A link relation with a single value that is a space-separated list of
items (NMTOKENS style) cannot be handled that way, because the
RFC6690-JSON/CBOR converter does not know which link relations use
NMTOKENS-style values.  That is sad, but I don't know a better way.
(Related to having "ct": "60".)

The formal syntax production indeed needs to be updated with this,
thanks for pointing that out!

Grüße, Carsten