Subject: Re: [core] Questions/Review on draft-ietf-core-resource-directory-12

From: Christian Amsüss <[email-masked]>
Date: Thu, 2 Nov 2017 10:27:12 +0100

Hello Jim,

responses inline:

On Tue, Oct 31, 2017 at 05:25:50PM -0700, Jim Schaad wrote:
> 1.  When creating a group registration, can the link to an endpoint be on a
> different resource directory?  This second RD could be either on a different
> machine or on the same machine (i.e. coap vs coaps).  If the answer is no,
> then do we need to think about some type of relation attribute in the case
> that multiple are listed somewhere?

I like to think so (having a more distributed model in mind), but we
haven't discussed it yet.

As for the specification, I think we should say that the CTs operating
on the tool working on the group "SHOULD refer to the endpoint
registrations either as a path-absolute relative reference (if it
obtained the reference from the Location option when it conducted the
registration itself), or use the reference as serialized when it
obtained the location from an endpoint lookup from the same RD origin".

This leaves the door open for federating RDs (where some endpoint
registrations are located at another RD host, but still available for
lookup at the closest host), but discourages blind mixing of
registration entities from unrelated RDs.

(Discussion about this can go continue on-list, but for tracking
purposes it's tracked at [85] too).

> 2.  Is there supposed to be an update/patch operation on groups or is this a
> replace operation ala what is done for endpoints?

Yes. Whatever will work for the more intricate resource link update
should also work for the more set-like group management.

Right now, groups can only be replaced by re-registering with the same
group name and a new set of links.

> 3. Are groups supposed to age out of existence?  Based on the current text I
> would say that this is a no.

So do I.

> 4.  Should have text on what happens to a group when an endpoint which ages
> out of existence.

Good point, taken up ([86]).

> 5.  When registering groups, is there a requirement that all of the
> endpoints be in the same domain as the group is?

That's the idea implied in the hierarchy, but probably answered better
by Michael or Peter who use the domain concept more than I do.

> 6.  In section 7.3 - First it would be better to be much more explicit on
> the set of items to be matched than the one statement that is current
> present.  From what is there, it is not clear to me that one would return a
> group if one of the resources of an endpoint in the group has the desired
> attribute.  However, based on everything else, it would appear to be true.
> Thus the search would be to go up the model (Figure 2) to all ancestor notes
> and down the tree to all descendent notes.  Is this a correct understanding?

That is a good point and an open issue[82], and indeed the
`/rd-lookup/gp?ep=node1` example contradicts the current text that only
covers going "up" in Figure 2.

I'm leaning strongly toward allowing both directions, ie. a query
parameter matches a candidate result entity if it matches on its
parameters or the link parameters of any of its containers or its
contents, transitively.

What worries me a bit is the performance impact of looking for possibly
matching resource attributes in any group lookup, but probably it
shouldn't.

> 7.  In section 7.3 - If doing a compound search criteria - Does all of the
> criteria need to be matched on ANY node in the search or ALL of the criteria
> on one node in the search?  Consider "?d=example.com&rt=foobar"  which would
> match the 'd' on the endpoint but the 'rt' on a resource.  So two questions
> - the first would be looking at two different items at two different levels
> (i.e. group and resource), the second would be two items at the same level
> (i.e. two different resources/endpoints which are children/grandchildren of
> me).

The quantifier sequence is "for each query parameter, there exists a
(suitably related) entity that matches the parameter". Each query
parameter can be satisfied by a different entity.

I don't fully understand the part after 'Consider', but
`?d=exmaple.com&rt=foobar` could give a single endpoint in an EP lookup
(which is the only endpoint to contain rt=foobar resources), but give
both its two rt=foobar resources in a resource lookup.

A query `?rt=foobar&rt=fnord` lookup could match on the endpoint lookup
interface (finding an endpoint that contains both a foobar and a fnord),
but might not give any result on the resource lookup interface if no
single resource has both types.

I'll try to make that clearer when addressing the abovementioned issue
of going up and down -- but did that answer your question?

Thank you for your comments
Christian

[82]: https://github.com/core-wg/resource-directory/issues/82
[85]: https://github.com/core-wg/resource-directory/issues/85
[86]: https://github.com/core-wg/resource-directory/issues/86

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Jim Schaad <[email-masked]>
Date: Thu, 2 Nov 2017 20:56:53 -0700

> -----Original Message-----
> From: Christian Amsüss [mailto:[email-masked]]
> Sent: Thursday, November 2, 2017 2:27 AM
> To: Jim Schaad <[email-masked]>
> Cc: [email-masked]; 'core' <[email-masked]>
> Subject: Re: Questions/Review on draft-ietf-core-resource-directory-12
> 
> Hello Jim,
> 
> responses inline:
> 
> > 1.  When creating a group registration, can the link to an endpoint be
> > on a different resource directory?  This second RD could be either on
> > a different machine or on the same machine (i.e. coap vs coaps).  If
> > the answer is no, then do we need to think about some type of relation
> > attribute in the case that multiple are listed somewhere?
> 
haven't
> discussed it yet.
> 
> As for the specification, I think we should say that the CTs operating on
the tool
> working on the group "SHOULD refer to the endpoint registrations either as
a
> path-absolute relative reference (if it obtained the reference from the
Location
> option when it conducted the registration itself), or use the reference as
> serialized when it obtained the location from an endpoint lookup from the
> same RD origin".
> 
lookup at the
> closest host), but discourages blind mixing of registration entities from
> unrelated RDs.
> 
> (Discussion about this can go continue on-list, but for tracking purposes
it's
> tracked at [85] too).

I really like the idea of being able to have entries in a group be on a
different server, right up to the point where I start trying to implement
the search algorithm that you have specified.  At that point my code is
suddenly making lots of potential queries to the other RD in order to find
out if some parent/child is going to have the attribute that I am looking
for.  

> > 4.  Should have text on what happens to a group when an endpoint which
> > ages out of existence.
> 
> That's the idea implied in the hierarchy, but probably answered better by
> Michael or Peter who use the domain concept more than I do.
> 
> > 6.  In section 7.3 - First it would be better to be much more explicit
> > on the set of items to be matched than the one statement that is
> > current present.  From what is there, it is not clear to me that one
> > would return a group if one of the resources of an endpoint in the
> > group has the desired attribute.  However, based on everything else, it
would
> appear to be true.
> > Thus the search would be to go up the model (Figure 2) to all ancestor
> > notes and down the tree to all descendent notes.  Is this a correct
> understanding?
> 
> That is a good point and an open issue[82], and indeed the `/rd-
> lookup/gp?ep=node1` example contradicts the current text that only covers
> going "up" in Figure 2.
> 
parameter
> matches a candidate result entity if it matches on its parameters or the
link
> parameters of any of its containers or its contents, transitively.
> 
shouldn't.

It's interesting, I was sure there was an example along the lines of

GET /rd-lookup/ep?rt=temperature-c

But I don't see it there now.

> 
> > 7.  In section 7.3 - If doing a compound search criteria - Does all of
> > the criteria need to be matched on ANY node in the search or ALL of
> > the criteria on one node in the search?  Consider
> > "?d=example.com&rt=foobar"  which would match the 'd' on the endpoint
> > but the 'rt' on a resource.  So two questions
> > - the first would be looking at two different items at two different
> > levels (i.e. group and resource), the second would be two items at the
> > same level (i.e. two different resources/endpoints which are
> > children/grandchildren of me).
> 
(suitably
> related) entity that matches the parameter". Each query parameter can be
> satisfied by a different entity.
> 
> (which is the only endpoint to contain rt=foobar resources), but give both
its
> two rt=foobar resources in a resource lookup.

It would be clearer if I said I was looking for a group or an end-point with
these parameters.  I probably should have used something other than d= to
something that was clearly only set on a group.

> interface (finding an endpoint that contains both a foobar and a fnord),
but
> might not give any result on the resource lookup interface if no single
resource
> has both types.
> 
> I'll try to make that clearer when addressing the abovementioned issue of
> going up and down -- but did that answer your question?

Yes, I think I understand the current desires.

Jim

---

From: 'Christian Amsüss' <[email-masked]>
Date: Fri, 3 Nov 2017 08:56:59 +0100

> out if some parent/child is going to have the attribute that I am looking

I was envisioning a cluster where each node keeps the data registered
with the other ndoes in an eventually consistent cache; maybe I'll find
time to sketch that out a bit further before the meeting.

> But I don't see it there now.

The example that's there is /rd-lookup/gp?ep=node1

> > > Consider "?d=example.com&rt=foobar"  which would match the 'd' on
> > > the endpoint but the 'rt' on a resource.  So two questions - the
> > > first would be looking at two different items at two different
> > > levels (i.e. group and resource), the second would be two items at
> > > the same level (i.e. two different resources/endpoints which are

I see. Both these questions (`WHERE ep.d=="example.com"` and `JOIN ...
WHERE group.d=="example.com"`) are, in the simple query language of
search parameters, expressed in the same way and returned as their
union.

We don't have a partitioning on the names to tell the intended query
apart. (d can be on both endpoint and group, and there's no registry for
link parameters so we can't rule out conflicts).

What we could do to mitigate that ambiguity is to make the query
language slightly more complext (so your endpoint queries might be
`?d=example.com&res→rt=foobar` vs. `?gp→d=example.com&res→rt=foobar`),
but that won't reduce the complexity but only make it more visible.

Do you think that the availability of search in both directions of the
tree is an issue for implementations (esp. distributed one), or is it
more a problem of specification or interface clarity that the meanig of
the query is not immediately obvious?

Best regards
Christian

---

From: peter van der Stok <[email-masked]>
Date: Sun, 05 Nov 2017 10:02:08 +0100

> by Michael or Peter who use the domain concept more than I do.

Sorry, a bit late in answering, for good reasons though.

My cut feeling is "no". Groups may be part of other domains than the 
contributing endpoints.
One might think of domain hierarchies.
e.g. endpoint in room domains, multicast in enveloping corridor domain.
I prefer to leave this to users of RD and not add extra rules.

Peter

---

From: peter van der Stok <[email-masked]>
Date: Mon, 06 Nov 2017 09:40:32 +0100

Hi Jim and Christian,

The discussion on a set of RDs is rather worrying to me.
I do understand the remark by Jim that the end-points composing a group 
could be stored on different RDs than the RD handling the group 
registration.
The RD is not aware of the remoteness of the eps and the client has to 
handle the distribution.
It is not different from registering anchors to different hosts.

However, envisaging a set of RDs implies many protocols to keep entries 
synchronized and handle communication errors.
I hope I misunderstood the intention of Christian.

Greetings,

Peter

>> on
>> purposes

---

From: Jim Schaad <[email-masked]>
Date: Mon, 6 Nov 2017 07:21:15 -0800

Peter,

In the current design it is easy for the RD to understand the remoteness of
an included endpoint, it would have a full URL to a different RD rather than
having a partial URL which resolves against the current RD.

That said, I am not in favor of the idea because of the searching problems.
I would rather just have a specialized synchronization API in a different
document.  (Which might be as easy as - I will publish any updates to the
following list of people.)

Jim

> From: peter van der Stok [mailto:[email-masked]]
> Sent: Monday, November 6, 2017 12:41 AM
> Cc: 'Christian Amsüss' <[email-masked]>; draft-ietf-core-
> [email-masked]; 'core' <[email-masked]>
> Subject: Re: [core] Questions/Review on
draft-ietf-core-resource-directory-12
> I do understand the remark by Jim that the end-points composing a group
could
> be stored on different RDs than the RD handling the group registration.
handle
> the distribution.
> >> > 1.  When creating a group registration, can the link to an endpoint
> >> > be on a different resource directory?  This second RD could be
> >> > either on a different machine or on the same machine (i.e. coap vs
> >> > coaps).  If the answer is no, then do we need to think about some
> >> > type of relation attribute in the case that multiple are listed
somewhere?
> >> option when it conducted the registration itself), or use the
> >> reference as serialized when it obtained the location from an
> >> endpoint lookup from the same RD origin".
> >>
> >> closest host), but discourages blind mixing of registration entities
> >> from unrelated RDs.
> >>
> > I really like the idea of being able to have entries in a group be on
> > a different server, right up to the point where I start trying to
> > implement the search algorithm that you have specified.  At that point
> > my code is suddenly making lots of potential queries to the other RD
> > in order to find out if some parent/child is going to have the
> > attribute that I am looking for.
> >

---

From: Christian M. Amsüss <[email-masked]>
Date: Fri, 10 Nov 2017 12:06:35 +0100

Hello Peter, hello Jim,

> The discussion on a set of RDs is rather worrying to me.

First, please see this whole RD federation topic as a matter of dancing
on thin ice to see whether it tolds. IMO, if RDs can conceptually be
federated, this shows that the whole linking thing works out well, but
if not, we should not change the specification to make it work (unless
we discover true inconsistencies from that thougth experiment).

This is nothing to be expected to work out-of-the box; just something
that happens to be easily doable because of how awesome RD is these days ;-)

I've added the example of what this could like (promised in an earlier
mail) below.

> following list of people.)

Do you think that'll benefit from being specified at all? The RD will be
an interaction point in heterogenous environments, but I doubt that
there will be much use for fully federated heterogenous RDs for some
time.

Best regards
Christian

---

A story of how federated RDs could work
=======================================

Say X and Y are federated RDs, Z is also federated but won't accept
registrations on its own. A, B and C are clients in close proxity to the
respective servers.

All clients behave only as per RD specification, with some tolerance to
non-local references during discovery.

Say C wants to register, and SLAAC gave it the address of Z:

| GET coap://Z/.well-known/core?rt=core.rd
| 2.05 Content
|   <coap://Y/rd>;rt="core.rd"

(because Z is just doing replication)

| POST coap://Y/rd?ep=node-C
|   </temp>;if="core.s";rt="temperature"
| 2.04 Created
| Location: /reg/1234

Since the beginning of federation, X and Z have been observing endpoints
and resources, and now get an update. The `only_local` thing is an RD
parameter registered for RD federation to prevents loops, and I'm
assuming a PATCH-style observation (even though it's not there yet):

| GET coap://Y/lookup/ep?only_local, Observe
| 2.06 Partial Content
|   New record </reg/1234>;ep="node-C";lt=84600

and on their second observation:

| GET coap://Y/lookup/res?only_local, Observe
| 2.06 Partial Content
|   New record </temp>;if="core.s";rt="temperature";anchor="coap://C"

If now any client asks X, it can get

| GET coap://X/lookup/res?rt=temperature
| 2.05 Content
|   </temp>;if="core.s";rt="temperature";anchor="coap://C"

without noticing anything, or it could look for all registered nodes
with core.s interfaces and get some local and some remote ones:

| GET coap://X/lookup/ep?if=core.s
| 2.05 Content
|   </reg/991>;ep="node-A",
|   </reg/1200>;ep="node-B",;anchor="coap://Y",
|   </reg/1234>;ep="node-C";anchor="coap://Y"

When RD Y prepares to shut down, it'll convey its federation membership
status of "retreating" to the other members (however tha's done), so Z
will no more announce its directory resource but the next closest one
(X). Later in that process, when C renews its membership, Y will cancel
it and respond 4.04, triggering C to retry the registration process,
find X, and get a new registrtion resource there. When all of Y's active
registrations have migrated or timed out, Y can quit the federation.

(Some RD endpoint implementations (shameless plug: aiocoap's) might be
robust enough to even tolerate Y silently going down if Z announced

| GET coap://Z/.well-known/core?rt=core.rd
| 2.05 Content
|   <coap://Y/rd>;rt="core.rd",
|   <coap://X/rd>;rt="core.rd"

; that won't do no harm, but as we don't specify any round-robin there,
it won't replace active monitoring of federated RDs).

-- 
When others do a foolish thing, you should tell them it is a foolish
thing. They can still continue to do it, but at least the truth is where
it needs to be.
  -- Dukhat