Subject: [core] Artart last call review of draft-ietf-core-links-json-07

From: Mark Nottingham <[email-masked]>
Date: Mon, 10 Apr 2017 20:49:47 -0700

Reviewer: Mark Nottingham
Review result: Ready with Issues

This specification is a relatively straightforward mapping of the
format described in RFC6690 (itself a serialisation of RFC5988bis
links) into JSON and CBOR. I don't have deep knowledge of CBOR, but
given the editorship of the document, I trust it's seen adequate
review in that regard.

The only potential issue is how this is achieved. Rather than defining
two new serialisations of RFC5988bis links (into JSON and CBOR), it
describes how to re-serialise RFC6690 documents into JSON and CBOR.
This means that any constraints upon RFC6690 documents are also
mirrored into these formats; e.g., the target IRI is constrained to be
a URI in 6690, and therefore can also only be a URI in JSON and CBOR,
despite these formats' ability to easily convey non-ASCII content.

In other words, the specification currently defines these link formats
in terms of the Link header (as defined in section 5 of RFC5988) --
along with all of the foibles of HTTP header syntax -- rather than
their abstract model (defined in Section 3).

Whether or not this is a problem depends on what's desired; if 6690 is
seen as effectively a profile of 5988, then it makes sense to express
it in those terms. If the full range of links capable of being
expressed in 5988 is desired, creating new serialisations of 5988
links (without a hop through 6690) is preferable.

If the current approach is kept, it'd be nice to clarify this
situation a bit in the Introduction.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Apr 2017 09:33:43 +0200

Hi Mark,

thank you a lot for another thoughtful review.

I don’t have an opinion yet on what we should do here, so I’ll supply some more background first.

The present spec is first and foremost intended to round-trip with RFC 6690, which is indeed stuck a bit on HTTP Link header field syntax.  However, the way this is used in practice is not meant to inherit the complexities of HTTP header field coding.  E.g., for CoRE Resource Directory and its DNS-SD compatibility, we want to represent a DNS-SD instance identifier (which is UTF-8) as a link attribute.  This is not a problem in JSON or CBOR, but also not in the way RFC 6690 is being used, i.e. as a UTF-8 based format.

I haven’t checked yet what 5988bis brings to the table and how to track this in 6690 or possibly a 6690bis.  There is no point in Big Web and Thing Web diverging here, so we should follow the lead.  But maybe that would touch both 6690 and the present document once 5988bis is done.

More importantly, we haven’t really put a lot of thinking into IRI support.  The CoAP data type “string” which is used for URI components such as Uri-Path is UTF-8, so we could embrace them by extending the rules in section 6 of RFC 7252 to cover IRIs.  I’m sure that, in practice, CoAP implementations already do this — it is not distinguishable in the CoAP packet whether the (decomposed) CoAP Uri components have been derived from URIs or IRIs.  But the metadata formats (6690 and its JSON/CBOR representations; various other JSON and CBOR based formats, e.g. from OCF) are still based on RFC 3986 URIs, except where they also use the CoAP decomposed form (e.g., CORAL).

While this is outside the scope of the CoRE WG, I personally would like to explore how link-format+cbor/+json could be useful for the Big Web.  If embracing IRIs there is all we need to make this happen, maybe that can be added with a warning that IRIs have to be converted back to URIs for 6690 link-format.

Grüße, Carsten

> On Apr 11, 2017, at 05:49, Mark Nottingham <[email-masked]> wrote:
> 
> _______________________________________________
> art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/art
>

---

From: Herbert Van de Sompel <[email-masked]>
Date: Tue, 18 Apr 2017 14:47:27 +0200

hi Mark, Carsten,

Sorry to be a bit late to the game.

Two things:

1. I would be very interested in documents serialized according to
application/link-format+json as described in the I-D, especially in the
context of an I-D Erik Wilde and I are authoring regarding Link Sets, see
https://datatracker.ietf.org/doc/draft-wilde-linkset-link-rel/

2. Regarding Mark's comment "This means that any constraints upon RFC6690
documents are also
mirrored into these formats": Mark mentions IRIs as a concern. I am also
concerned about the rules for interpretation of (the Context IRI of) links
as described in Section 2.1 of 6690 (
https://tools.ietf.org/html/rfc6690#section-2.1). It seems to me that these
also introduce constraints that go beyond 5988. I may be mistaken with that
regard because I have never fully understood that section of 6690 (i.e. the
use of "base URI", "origin", "Context URI"). But, when compared to 5988,
the section comes across as imposing constraints that are intended to allow
the straightforward use of relative URIs in constrained environments as a
means to decrease the payload. If my interpretation is correct, then I
would very much favor spec-ing the json link format in terms of 5988 rather
than 6690.

Cheers

Herbert Van de Sompel

On Tue, Apr 11, 2017 at 9:33 AM, Carsten Bormann <[email-masked]> wrote:

> The present spec is first and foremost intended to round-trip with RFC
> 6690, which is indeed stuck a bit on HTTP Link header field syntax.
> However, the way this is used in practice is not meant to inherit the
> complexities of HTTP header field coding.  E.g., for CoRE Resource
> Directory and its DNS-SD compatibility, we want to represent a DNS-SD
> instance identifier (which is UTF-8) as a link attribute.  This is not a
> problem in JSON or CBOR, but also not in the way RFC 6690 is being used,
> i.e. as a UTF-8 based format.
>
> I haven’t checked yet what 5988bis brings to the table and how to track
> this in 6690 or possibly a 6690bis.  There is no point in Big Web and Thing
> Web diverging here, so we should follow the lead.  But maybe that would
> touch both 6690 and the present document once 5988bis is done.
>
> More importantly, we haven’t really put a lot of thinking into IRI
> support.  The CoAP data type “string” which is used for URI components such
> as Uri-Path is UTF-8, so we could embrace them by extending the rules in
> section 6 of RFC 7252 to cover IRIs.  I’m sure that, in practice, CoAP
> implementations already do this — it is not distinguishable in the CoAP
> packet whether the (decomposed) CoAP Uri components have been derived from
> URIs or IRIs.  But the metadata formats (6690 and its JSON/CBOR
> representations; various other JSON and CBOR based formats, e.g. from OCF)
> are still based on RFC 3986 URIs, except where they also use the CoAP
> decomposed form (e.g., CORAL).
>
> While this is outside the scope of the CoRE WG, I personally would like to
> explore how link-format+cbor/+json could be useful for the Big Web.  If
> embracing IRIs there is all we need to make this happen, maybe that can be
> added with a warning that IRIs have to be converted back to URIs for 6690
> link-format.
>
> Grüße, Carsten
>
>

-- 
Herbert Van de Sompel
Digital Library Research & Prototyping
Los Alamos National Laboratory, Research Library
http://public.lanl.gov/herbertv/
http://orcid.org/0000-0002-0715-6126

==

---

From: Erik Wilde <[email-masked]>
Date: Mon, 24 Apr 2017 10:49:03 +0200

hello.

just adding my voice here and since i am collaborating with herbert on 
this, my position is relatively clear.

On 2017-04-18 14:47, Herbert Van de Sompel wrote:
> 2. Regarding Mark's comment "This means that any constraints upon
> RFC6690 documents are also
> concerned about the rules for interpretation of (the Context IRI of)
> links as described in Section 2.1 of 6690
> (https://tools.ietf.org/html/rfc6690#section-2.1). It seems to me that
> these also introduce constraints that go beyond 5988. I may be mistaken
> with that regard because I have never fully understood that section of
> 6690 (i.e. the use of "base URI", "origin", "Context URI"). But, when
> compared to 5988, the section comes across as imposing constraints that
> are intended to allow the straightforward use of relative URIs in
> constrained environments as a means to decrease the payload. If my
> interpretation is correct, then I would very much favor spec-ing the
> json link format in terms of 5988 rather than 6690.

very much agreed, it would be great to have a generic way of serializing 
links as standalone resources (ideally in a way that is able to preserve 
their context, so that relative URIs are well-defined). my concern is 
that RFC 6690 was not intended to do this (it adds constraints to RFC 
5988), and thus draft-ietf-core-links-json has the same limitations.

to be fair, the draft does not intend to define a general-purpose web 
link serialization, it is simply intended as a serialization of the 
specialized model defined in RFC 6690.

i am not sure what the best way forward is. the representations defined 
in this draft are tantalizingly close to general-purpose serializations 
of RFC 5988. but the draft is clearly intended to be (one more) building 
block in the "CoRE-only" universe. two suggestions:

- add language that makes it clear that because of the limitations of 
RFC 6690, the media types in this draft should not be considered as 
general-purpose serializations of web links.

- consider adding a serialization of web links to RFC 5988bis. this 
would address the problem of how to serialize web links outside of the 
HTTP link header field.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked] |
            | http://dret.net/netdret    |
            | http://twitter.com/dret    |

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Apr 2017 13:52:15 +0200

On Apr 24, 2017, at 10:49, Erik Wilde <[email-masked]> wrote:
> 
> - consider adding a serialization of web links to RFC 5988bis. this would address the problem of how to serialize web links outside of the HTTP link header field.

Sounds good to me.

What needs to be done to make sure links-json is a proper subset of this?

Grüße, Carsten

---

From: Erik Wilde <[email-masked]>
Date: Mon, 24 Apr 2017 14:48:29 +0200

> What needs to be done to make sure links-json is a proper subset of this?

hard to tell as "this" is just speculation at this point. generally 
speaking, i don't think it's such a great idea to piggyback on standards 
and then reduce their expressiveness. imho that's one of the well-known 
anti-patterns of interop: (extended) subsets.

but if you're shooting for a subset i think that's where you are right now.

it would be better to make sure that serializations of web links 
actually can represent web links and not just some of the information 
that they convey. that train may have left the station with RFC 6690, 
but maybe for the JSON and CBOR serializations that can be changed.

cheers,

dret.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Apr 2017 14:55:56 +0200

On Apr 24, 2017, at 14:48, Erik Wilde <[email-masked]> wrote:
> hard to tell as "this" is just speculation at this point. generally speaking, i don't think it's such a great idea to piggyback on standards and then reduce their expressiveness. imho that's one of the well-known anti-patterns of interop: (extended) subsets.
> 
> it would be better to make sure that serializations of web links actually can represent web links and not just some of the information that they convey. that train may have left the station with RFC 6690, but maybe for the JSON and CBOR serializations that can be changed.

Right.  Can you be more specific what you would want to see here?

Grüße, Carsten

---

From: Erik Wilde <[email-masked]>
Date: Tue, 25 Apr 2017 14:35:54 +0200

hello carsten.

> Right.  Can you be more specific what you would want to see here?

two possibilities:

- to do things well it would be better to have web link serializations 
that cover *all* of RFC 5988 (bis). that's a hard thing to do and will 
take a while.

- for the RFC 6690-based variants under consideration right now, it 
would be helpful to very explicitly point out that they are *not* 
general-purpose serializations of web links, but instead inherit the 
limitations of the underlying spec.

cheers,

dret.

---

From: Herbert Van de Sompel <[email-masked]>
Date: Tue, 25 Apr 2017 14:50:15 +0200

On Tue, Apr 25, 2017 at 2:35 PM, Erik Wilde <[email-masked]> wrote:

>> it would be better to make sure that serializations of web links actually
>>> can represent web links and not just some of the information that they
>>> convey. that train may have left the station with RFC 6690, but maybe for
>>> the JSON and CBOR serializations that can be changed.
>>>
> that cover *all* of RFC 5988 (bis). that's a hard thing to do and will take
> a while.
>
>
As Erik previously indicated, it would be great if this could be done as
part of RFC5988bis.

> - for the RFC 6690-based variants under consideration right now, it would
> be helpful to very explicitly point out that they are *not* general-purpose
> serializations of web links, but instead inherit the limitations of the
> underlying spec.
>
>
That would, indeed, be good. But, in case RFC5988bis would spec a (JSON)
serialization, it seems to me that it would be rather helpful for the CORE
community if the RFC 6690 JSON serialization would be based on it.

Cheers

herbert

==

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Apr 2017 15:06:48 +0200

> On Apr 25, 2017, at 14:50, Herbert Van de Sompel <[email-masked]> wrote:
> 
> - to do things well it would be better to have web link serializations that cover *all* of RFC 5988 (bis). that's a hard thing to do and will take a while.
> 
> 
> As Erik previously indicated, it would be great if this could be done as part of RFC5988bis.

Yes.

But covering all of RFC 5988 is mainly hard because of the vagaries of HTTP header field encoding.
We don’t have that problem in RFC 6690, so it is easy to have RFC 6690 and links-json as a proper subset if we want to.

>  - for the RFC 6690-based variants under consideration right now, it would be helpful to very explicitly point out that they are *not* general-purpose serializations of web links, but instead inherit the limitations of the underlying spec.
> 
> 
> That would, indeed, be good. But, in case RFC5988bis would spec a (JSON) serialization, it seems to me that it would be rather helpful for the CORE community if the RFC 6690 JSON serialization would be based on it. 

I would hope so, but remember that links-json is already out there.

The RFC 6690 JSON serialization is pretty much a no-brainer, and there are no obvious ways to do things very different (well, you could choose a different name for “href”, but that would not be very bright).   CBOR adds a few bike sheds, but these can be painted in the same color with little effort.

If a superset RFC 5988 JSON serialization deviates from this (i.e., is not a proper superset), I think this would require some willful effort.
(I’m happy to be proven wrong; that’s why I was asking if any specifics are known here already.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Apr 2017 18:46:24 +0200

On 2017-04-25 15:06, Carsten Bormann wrote:
 > ...

Do you have a specific problem in mind?

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Apr 2017 20:59:59 +0200

> Do you have a specific problem in mind?

This work was done half a decade ago (May 18, 2012, that is), so I don’t remember all the details.
RFC 6690 says:

   In
   order to convert an HTTP Link Header field to this link format, first
   the "Link:" HTTP header is removed, any linear whitespace (LWS) is
   removed, the header value is converted to UTF-8, and any percent-
   encodings are decoded.

So we get rid of all that fun before it becomes RFC 6690 (and CoAP of course is all UTF-8 in any case).

So far, we have run into one real case where that approach is a limitation, and that is in URIs:

The link

coap://example.com?stupid%3Dkey=4711

is not distinguishable from

coap://example.com?stupid=key=4711

(The typical reaction of an implementer is “then don’t do that!” [1,2].)

We know that because the CoAP protocol itself also completely runs in the UTF-8 domain (there is no percent encoding on the wire); I’m not sure that simplification actually hurt in RFC 6690 use cases yet.

(Note that it is rather likely to see user-visible strings in DNS-SD instance names, which turn into “ins” link attributes, so this is not only a URI/IRI/WVUS [3] issue.)

We don’t have a lot of experience with lingering legacy charsets in HTTP headers in the constrained space, so I’m sure there are other side effects.

Grüße, Carsten

[1]: http://www.condenaststore.com/-sp/Then-don-t-do-that-New-Yorker-Cartoon-Prints_i14788077_.htm
[2]: http://www.catb.org/~esr/jargon/html/D/Don-t-do-that-then-.html
[3]: https://url.spec.whatwg.org/#valid-url-string (WHATWG Valid URL String, or WVUS for short)

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 25 Apr 2017 13:09:40 -0700

> On Apr 25, 2017, at 11:59 AM, Carsten Bormann <[email-masked]> wrote:
> 

Well, that's broken.

> (The typical reaction of an implementer is “then don’t do that!” [1,2].)

That isn't a "limitation".  It's a bug to decode pct-encoded octets in
a URI before decomposing the reference into its parts.  ASCII is already
in UTF-8.  Decoding a pct-encoding doesn't make it "more UTF-8"; it just
means the string is no longer a URI reference.  That's broken.  So utterly
broken that it obviously wasn't reviewed by the right people.

> We know that because the CoAP protocol itself also completely runs in the UTF-8 domain (there is no percent encoding on the wire); I’m not sure that simplification actually hurt in RFC 6690 use cas…

*sigh*

....Roy

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 25 Apr 2017 23:25:48 +0200

>>  removed, the header value is converted to UTF-8, and any percent-

OK, let me start typing that errata report then.

> That isn't a "limitation”.  

For RFC6690 users, it pretty much is, because certain URIs don’t work.
They tend to design their URIs in such a way that they do, probably more so because these designs are natural for them than because they are fully aware of that limitation.

> It's a bug to decode pct-encoded octets in
> a URI before decomposing the reference into its parts.  

Well, percent-encoding is playing two roles in RFC 3986: hiding characters within syntactic elements from their delimiter roles, and encoding non-ASCII (and C0 etc.) characters.
The passage I cited from RFC 6690 got nicely rid of the latter, and broke the former(*).

> ASCII is already

So what should I write into the errata report?

Or more generally speaking, how should we fix RFC 6690, without creating a need for constrained nodes to do full URI processing?

Maybe it is sufficient to document the limitation in the errata, for now?

And, more to the point of the subject line, how should we handle this on the JSON/CBOR level?

There definitely will be a round-tripping problem with RFC 6690 if the URIs collide with the above limitation of RFC 6690.  But that’s OK because that defines the subset.

To be more general, not doing any percent-decoding of URIs when creating JSON/CBOR from scratch is probably the easy way, but it means that when we want to phase out RFC 6690 on the constrained level by replacing it with JSON/CBOR, there is additional complexity.  Horribile dictu, but maybe IRIs are the right thing to do here.

Grüße, Carsten

(*) It may be worth pointing out that the amount of breakage here is much larger than for CoAP itself, which does the percent-decoding only after decomposing a URI into what CoAP considers to be its components, so the URI parsing works properly — coap://example.com/foo%2fbar has one path segment, “foo/bar”.
But the application semantics of hiding application delimiters, which my example above is breaking, is not supported in CoAP either.
Some people think that URIs should be carried around in that decomposed form throughout the constrained space, and I can’t blame them.
I don’t have data how many URI libraries in active use in the non-constrained space get this particular detail right, either.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 26 Apr 2017 17:51:45 +0200

> OK, let me start typing that errata report then.

Below is a draft errata report.

Is this information correct?
Is it sufficient?

Obviously, this errata report doesn’t by itself answer the important questions raised about links-json, but it might be a useful outcome of this discussion anyway.

Grüße, Carsten

Report Errata for RFC6690

Date:	2017-04-26
Name:	Carsten Bormann
Email:	[email-masked]
Type:	Editorial
Section:	2

Original Text:

   [...] In

Corrected Text:

   (add after unchanged original text:)

   Note that this percent-decoding damages URIs that percent-encode
   reserved characters (i.e., characters out of ":/?#[]@!$&'()*+,;=",
   not including the double quotes).  Such URIs therefore generally
   cannot be successfully used with RFC 6690 link-format.

Notes:

   Fully percent-decoding URIs before placing them into the
   link-format reduces complexity in processing link-format, but
   creates a limitation on the set of URIs that link-format faithfully
   can represent.  This may not be as widely known as is desirable,
   creating a pitfall for unwitting users of RFC 6990.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 26 Apr 2017 14:37:40 -0700

> On Apr 26, 2017, at 8:51 AM, Carsten Bormann <[email-masked]> wrote:
> 

No.  An RFC that isn't interoperable (in this case, because it
contradicts its own normative references) is a design error.

I don't agree with the entire contents of RFC6690.  The abstract alone is
enough for me to drop it in the nearest recycling bin; the design has
nothing to do with REST (an architectural style, not an architecture).

Nevertheless, the error (a non-interoperable design error) in RFC6690 is in
section 4.1.  It is either a bad URI template (if the examples are correct)
or a bad description of how to process that template.

  /.well-known/core{?search*}

RFC6570 requires that the above template's values be pct-encoded during
expansion, so the examples given in 4.1:

  The following are examples of valid query URIs:

  o  ?href=/foo matches a link-value that is anchored at /foo

  o  ?href=/foo* matches a link-value that is anchored at a URI that
     starts with /foo

  o  ?foo=bar matches a link-value that has a target attribute named
     foo with the exact value bar

  o  ?foo=bar* matches a link-value that has a target attribute named
     foo, the value of which starts with bar, e.g., bar or barley

  o  ?foo=* matches a link-value that has a target attribute named foo

are almost all wrong.  They would have to be

  The following are examples of valid query URIs:

  o  ?href=%2Ffoo matches a link-value that is anchored at /foo

  o  ?href=%2Ffoo%2A matches a link-value that is anchored at a URI that
     starts with /foo

  o  ?foo=bar matches a link-value that has a target attribute named
     foo with the exact value bar

  o  ?foo=bar%2A matches a link-value that has a target attribute named
     foo, the value of which starts with bar, e.g., bar or barley

  o  ?foo=%2A matches a link-value that has a target attribute named foo

Alternatively, if the examples are correct, the specification needs to replace
the above URI template with a more accurate list of such templates to match
the examples:

  /.well-known/core
  /.well-known/core?href={+reference}
  /.well-known/core?anchor={+anchor}
  /.well-known/core?type={+mt}
  /.well-known/core{?rel}
  /.well-known/core{?rev}
  /.well-known/core{?hreflang}
  /.well-known/core{?media}
  /.well-known/core{?title}
  /.well-known/core{?rt}
  /.well-known/core{?if}
  /.well-known/core{?sz}
  /.well-known/core{?link-extension*}
  /.well-known/core?href={+reference}*
  /.well-known/core?anchor={+anchor}*
  /.well-known/core?type={+mt}*
  /.well-known/core{?rel}*
  /.well-known/core{?rev}*
  /.well-known/core{?hreflang}*
  /.well-known/core{?media}*
  /.well-known/core{?title}*
  /.well-known/core{?rt}*
  /.well-known/core{?if}*
  /.well-known/core{?sz}*
  /.well-known/core{?link-extension*}*

... [note that this still has an ambiguity problem for href string
    values that happen to end in "*"]

and then the text in 4.1 should be fixed as well:

Original Text:

  Value Strings are percent-decoded
  ([RFC3986], Section 2.1) before matching; similarly,

Corrected Text:

  Value Strings for the href, anchor, and type attributes (reserved templates)
  are not percent-decoded ([RFC3986], Section 2.1) before matching;
  for all other templates, Value Strings are percent-decoded only once
  before matching; note that a value which originally contained a
  percent-encoded triplet of "%20" would be encoded as "%2520" by a
  non-reserved template expansion and restored here as "%20".  Similarly,

The above would make RFC6690 consistent with its normative references.

A more sensible design would separate the response format (which is fine
as a media type) from the query syntax (which doesn't need to be limited
to link attributes anyway).

....Roy