Subject: [core] draft-ietf-core-coap-09 - Proxying of multicast requests

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 17 Apr 2012 11:14:54 +0000

In Section 5.7 (Proxying) the possibility of a multicast request in the Proxy-Uri  is not explicitly considered.
Should we mention this possibility? (If yes, do we need to update/adapt our caching text to include the multicast case?)

Current text suggests that only unicast has been considered, e.g. single destination and single stored response mentioned:

"If the proxy does not employ a cache, then it simply forwards the
   translated request to the determined destination. Otherwise, if it
   does employ a cache but does not have a stored response that matches
   the translated request and is considered fresh, then it needs to
   refresh its cache according to Section 5.6."

"Otherwise, the proxy returns the response to the client."

If multicast is supported then this section may have to be updated, OR we could add some sentences only applicable for the multicast case.
See also the email thread that ended with this message http://www.ietf.org/mail-archive/web/core/current/msg02802.html
kind of concluding that we do not specify internal processing (filtering, aggregation, etc.) by an intermediary for multicast proxying. Though caching may still be in scope of core-coap.

regards,
Esko

________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 17 Apr 2012 17:40:04 +0200

Good point.

I think the most important thing to understand here is how a multicast request (there are no multicast responses) uses/modifies our endpoint abstraction.

Say, I do a 

	GET coap://[ff02::1]/.well-known/core

I get back responses from a number of endpoints, including, say [aaaa::4711].
Is that response cacheable in the same way as a response to

	GET coap://[aaaa::4711]/.well-known/core

would be?  Does it replace that entry, if I have it?
Probably not, because we have some rules that make servers respond differently when they are addressed by multicast.  (In a way, sending a request via multicast is invoking an implicit critical option.)

So what would we cache?

(Clearly, the response can't be cached as the response for

	GET coap://[ff02::1]/.well-known/core

first of all, there may be many of these, and the responding set may change all the time.)

So maybe we cache it as a 

	GET coap://[aaaa::4711]/.well-known/core

with a special "virtual" critical option "this was a response to multicast"?

Grüße, Carsten

---

From: "Dijk, Esko" <[email-masked]>
Date: Thu, 26 Apr 2012 12:10:33 +0000

> with a special "virtual" critical option "this was a response to multicast"?

Maybe the "virtual critical option" explanation is ok, or maybe it need not be introduced as a concept by mandating that cached multicast responses MUST NOT be used to serve unicast requests and vice versa.

Should we then mention in the draft that multicast requests address a different resource namespace (similar to coaps requests being different from coap requests)?
Whether this is the case or not is not fully clear to me yet from the text. Or maybe it should be the same resource namespace with just some added rules for when to respond and when not.

groeten,
Esko

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]]
Sent: Tuesday 17 April 2012 17:40
To: Dijk, Esko
Cc: [email-masked] WG
Subject: Re: [core] draft-ietf-core-coap-09 - Proxying of multicast requests

Good point.

Say, I do a

So what would we cache?

Grüße, Carsten

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Thu, 26 Apr 2012 14:37:50 +0200

> with a special "virtual" critical option "this was a response to multicast"?

How can we reuse this response?

case A) We can use it to respond to successive unicast requests.

Then the cache updates the older entry (if any).

case B) We cannot reuse it to respond to successive unicast requests.

Multicast responses should be cached separately.

###### PROBLEMS WITH CASE B
In case B there is (at least) one issue, which is briefly discussed in
Sec. 4.3.3 of http-mapping I-D
(http://tools.ietf.org/html/draft-castellani-core-http-mapping-03#section-4.3.3).
I synthetize the problem afterwards.

When a new multicast request cames in, generally speaking the proxy
couldn't rely only on cached responses to serve client request, since
multicast group membership could be dynamic, and the cached
representation(s) may represent a subset of the actual group, due to
network loss of the previous request/response exchange.

The proxy can, in general, effectively reuse cached representation(s)
only when it has an updated knowledge of the hosts partecipating to
the multicast group, if something is missing the request should be
repeated anyway.
#######

However if we are satisfied with case A, multicast itself could be a
very interesting approach to update the proxy cache for popular
resources, and could even be paired with Observe to obtain this in a
very efficient way.

Best,
Angelo

---

From: [email-masked]
Date: Thu, 26 Apr 2012 15:50:15 +0200

>Should we then mention in the draft that multicast requests address a 
different resource namespace (similar to coaps requests being different 
from coap requests)?
>Whether this is the case or not is not fully clear to me yet from the 
text. Or maybe it should be the same resource namespace with just some 
added rules for when to respond and when not.

For me it's clear that the resource trees could be different between 
unicast and multicast requests. But if we register an AllCoapServers 
multicast address we could make Carsten's rule applicable to this special 
adddress
GET coap://[AllCoapServers]/.well-known/core == GET 
coap://[aaaa::4711]/.well-known/core

We could also use Location-* options in responses to multicast requests to 
indicate where is located the corresponding unicast resource. If the 
multicast address is not AllCoapServers and Location-* options are not 
recognized we can just say that the response is not cacheable.

My 2 cents
Matthieu

---

From: "Dijk, Esko" <[email-masked]>
Date: Thu, 26 Apr 2012 13:59:34 +0000

Suppose a proxy has no knowledge of hosts participating in a multicast group. Then indeed it can't rely on its cached multicast responses alone. I was thinking more about this case and how it can still use cached responses:

 1- proxy caches previous responses to a multicast request
 2- for each new Proxy-Uri multicast request that comes in, it always sends out a CoAP multicast request to the multicast group
   [since there may be new group members that joined meanwhile or ones that did not get the previous request]
 3- proxy caches all incoming responses, possibly overwriting the older stored responses
 4- invalid / expired responses are of course removed from the cache
 5- proxy sends all responses (both cached-still-fresh and 'new') back to the original client.

Here caching helps to get a more complete set of responses: if for some reason 20% of group members don't receive a 2nd multicast request; the proxy may still have valid information in cache about all or most of these 20% nodes which was received during a 1st multicast CoAP request.

Note in the above case an implicit assumption is that a CoAP server, who sends out a cacheable response in response to a multicast request, expects/promises to be part of that multicast group for at least the time (Max-Age) of the response. That could be written explicitly (if we want to say something about caching of responses to multicast requests).

Aren't there 4 cases by the way?

        |  reuse for multicast Y/N
--------+----------------------------
reuse   |   Y/Y         Y/N
for     |
unicast |   N/Y         N/N
Y / N   |

regards
Esko

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Angelo P. Castellani
Sent: Thursday 26 April 2012 14:38
To: Carsten Bormann
Cc: Dijk, Esko; [email-masked] WG
Subject: Re: [core] draft-ietf-core-coap-09 - Proxying of multicast requests

Best,
Angelo

---

From: Klaus Hartke <[email-masked]>
Date: Thu, 26 Apr 2012 17:06:39 +0200

>  4- invalid / expired responses are of course removed from the cache

If a response has an etag, the cache can attempt to revalidate an
expired response instead of immediately removing it. How would
revalidation work with multicast? Etags are resource-local
identifiers, and the etag generated by one origin server might
identify a completely different resource state on another server.

>        |  reuse for multicast Y/N

Yes.

So, combining what Esko and Matthieu say, what about something along
the following lines:

* The responses received in reply to a GET request to a multicast
group can be used to satisfy a subsequent request on the same request
URI, but can not be validated using the ETag option.

* Before the stored responses can be used to satisfy the subsequent
request, the proxy always makes a new request to the multicast group
and updates the cache with the received responses.

* A server MAY include a Content-Location option in a response to a
GET request. Similar to the HTTP Content-Location header field, the
option indicates that

    if one were to perform a GET on this URI at the time
    of this message's generation, then a 2.05 (Content)
    response would contain the same representation that
    is enclosed as payload in this message.

* If a response contains a Content-Location option, then a cache MAY
use the response to satisfy a subsequent request on the
Content-Location URI in addition to the request URI. (This is
different from HTTP.)

* If a response contains both an ETag option and a Content-Location
option, then a cache can revalidate the response by making a GET
request on the Content-Location URI.

The more the use of multicast modifies the normal rules of coap
though, the more I'd like to see everything multicast-related in a
different document together rather than sprinkled all over the core
document.

Klaus

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Thu, 26 Apr 2012 17:14:10 +0200

> and updates the cache with the received responses.

In this way, when network loss is low enough, there is no benefit at
all in having that cache.

As said in my previous message, the only "real" use-case in caching
multicast responses is to serve the responses in unicast requests.

Best,
Angelo

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 26 Apr 2012 17:37:21 +0200

On Apr 26, 2012, at 17:06, Klaus Hartke wrote:

> can not be validated using the ETag option

That would be a serious loss.

I think we have to keep in mind why we are using multicast:
Do discover servers that have useful content.
Once we have discovered them, we might want to switch to unicast to interact with them in more detail.  It should be possible to use previous content received via multicast as a basis of such a unicast relationship.

So, with respect to the "virtual option" this-was-obtained-via-a-multicast-request:
This should not cause a new document space or a new namespace for Etags (like the difference between http:// and https:// causes, which I think simply is a bug of HTTP that we don't need to import into CoAP).
I should be able to use an Etag received via a unicast response to a multicast request, in order to validate the representation in a unicast request to the same resource.

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Thu, 26 Apr 2012 19:08:45 +0300

> I should be able to use an Etag received via a unicast response to a multicast request, in order to validate the representation in a unicast request to the same resource.

+1

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Klaus Hartke <[email-masked]>
Date: Thu, 26 Apr 2012 18:10:53 +0200

> I should be able to use an Etag received via a unicast response to a multicast request, in order to validate the representation in a unicast request to the same resource.

Yes. That's what I tried to enable with the Content-Location option,
because you need the unicast request URI to validate the response. But
maybe the option is not really needed...

Second attempt:

* The responses received in reply to a GET request to a multicast
group MAY be used to satisfy a subsequent request on the same request
URI.

* When a new Proxy-Uri multicast request comes in, the proxy always
makes a new request to the multicast group and MAY update the cache
with the received responses.

* A response received in reply to a GET request to a multicast group
MAY also be used to satisfy a subsequent request on the unicast
request URI. The unicast request URI is obtained by replacing the
authority part of the request URI with the transport layer source
address of the response message.

* A cache MAY revalidate a response by making a GET request on the
unicast request URI.

* A GET request to a multicast group MUST NOT contain an ETag option.

Klaus

---

From: "Dijk, Esko" <[email-masked]>
Date: Fri, 27 Apr 2012 07:05:02 +0000

+1

I am too in favor of the "shared resource namespace" and I think this formulation also addresses Angelo's concerns.

Esko

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Klaus Hartke
Sent: Thursday 26 April 2012 18:11
To: [email-masked]
Subject: Re: [core] draft-ietf-core-coap-09 - Proxying of multicast requests

Second attempt:

Klaus
_______________________________________________
core mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Fri, 27 Apr 2012 09:22:07 +0200

> * A cache MAY revalidate a response by making a GET request on the

What are we standardizing here?

If we decide that the representation provided in response to a
multicast request MUST be the same as in the unicast request, then it
is obviously the case to cache those representations and we open the
path for another efficient cache update mechanism.

But if we don't enforce that this by a MUST, a couple of MAYs don't
provide any standard at all, everything is lecit and nothing is
wrong... Is it really useful?

> * A GET request to a multicast group MUST NOT contain an ETag option.

This is the only rule provided, which I am not sure that I share.

EXAMPLE: Imagine a resource that has the same representation
everywhere (e.g. a boolean one), and you want to check that its
representation is still the same everywhere. Why shouldn't you use
Etag?

Angelo

---

From: [email-masked]
Date: Fri, 27 Apr 2012 09:24:53 +0200

>address of the response message.

In concrete terms it means we can't create virtual servers with multicast 
addresses without also having a new unicast address attached to this 
virtual server (impossible with IPv4 we need names and DNS for that).
I can see use cases where a multicast virtual server that serves a sub 
tree of the main resource tree could be useful. That's why I liked the 
idea of having the same multicast/unicast resource tree only for the 
AllCoapServers multicast address.

Matthieu

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 27 Apr 2012 09:39:33 +0200

> with the received responses.

One thing we are missing to make this more useful is a way to suppress responses the client already has.
For unicast, we do some of this with ETags, but these are in a namespace made up by each server, so they don't mean anything for a multicast group as a whole.
For multicast, we'd have to list pairs of authorities and ETags.
Obviously, these lists can become big quickly.
There are good ways to do suppression on a probabilistic basic (bloom filters are the obvious way, which can be asymptotically perfect in particular if you vary some hash seed with each request).

Given the limited use of multicast in constrained networks, I'm not suggesting that we standardize anything here, but we could add a note that this spec does not yet define a suppression mechanism ("left for further study" would be the CCITT term :-).

(Then some bright young grad students can go ahead and spend a year to design the perfect bloom filter for constrained node/networks...)

Note that we also employ the concept of filters already (for link-format), so if we are just searching, say, for lights that are still on, we know how to do that.

Grüße, Carsten