Subject: [core] TCP, UDP, Proxy and Observe

From: Jim Schaad <[email-masked]>
Date: Mon, 24 Apr 2017 09:19:17 -0700

I have been sketching out what my TCP/UDP and UDP/TCP proxy code would look
like and I am having a problem deciding what should be done for the observe
option.

If you have the case of Client->UDP->Proxy->UDP->Server, then dealing with
out of order observations can be dealt with in the following ways:

1. Pass them through unchanged or subtract a constant from the observer
value.  This pushes the problem of out of order observes onto the client.
The proxy would save the high water content.
2. Filter on observe value so that out of order transmissions are
suppressed.
3. Keep out of order items around for a while and see if the out of order
items show up and transmit them back as in order.  I missing items do not
show up until the time out, then suppress the out of order items.
4. Reorder the numbers so that things are seen in the wrong order by the
client.  (Obviously a wrong answer.)

I assumed that 1 was the correct answer and deal with things that way.

Now I come to the question of Client->TCP->Proxy->UDP->Server.  Since the
observe value is now to be ignored or removed, the first option above will
no longer work and I am not sure which the remaining options is the correct
one to do.  Does a reliable transmission protocol mean that we should
suppress items?

Jim

---

From: Robert Quattlebaum <[email-masked]>
Date: Mon, 24 Apr 2017 09:45:37 -0700

> On Apr 24, 2017, at 9:19 AM, Jim Schaad <[email-masked]> wrote:

According to the data model that observe is based on, out-of-order observe response packets are considered to be out-of-date and thus should be ignored. Proxies must be aware of outstanding observations and keep track of the observe counters to make sure that out-of-order packets are dropped. I'm not super happy about the mechanism, but that seems to be what the standard has settled on. Passing everything through and letting the client sort it out is explicitly not allowed.

-- RQ

> On Apr 24, 2017, at 9:19 AM, Jim Schaad <[email-masked]> wrote:
> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Apr 2017 18:54:20 +0200

On Apr 24, 2017, at 18:45, Robert Quattlebaum <[email-masked]> wrote:
> 
> According to the data model that observe is based on, out-of-order observe response packets are considered to be out-of-date and thus should be ignored.

Right, Observe gives you eventual consistency, where data that is not fresh is not of interest.

Grüße, Carsten

---

From: Klaus Hartke <[email-masked]>
Date: Mon, 24 Apr 2017 19:00:31 +0200

> The proxy would save the high water content.

The sequence numbers in notifications are not end-to-end; they are
only meaningful between each pair of hops; each hop has to generate
its own sequence numbers [1]. It might be possible for a hop to
generate sequence numbers based on the values it receives, but my head
hurts when thinking about temporal sequence numbers, so I wouldn't
recommend it.

>From Section 1.3 [2]:

   o  It follows a best-effort approach for sending the current
      representation to the client after a state change: clients should
      see the new state after a state change as soon as possible, and
      they should see as many states as possible.

So, depending on whether you optimize for "as soon as possible" or "as
many as possible", you should do 2 or 3. I would recommend 2.

This applies equally to reliable and unreliable transports.

Klaus

[1] https://tools.ietf.org/html/rfc7641#section-5
[2] https://tools.ietf.org/html/rfc7641#section-1.3

---

From: Jim Schaad <[email-masked]>
Date: Mon, 24 Apr 2017 10:26:22 -0700

-----Original Message-----
From: Robert Quattlebaum [mailto:[email-masked]] 
Sent: Monday, April 24, 2017 9:46 AM
To: Jim Schaad <[email-masked]>
Cc: core <[email-masked]>
Subject: Re: [core] TCP, UDP, Proxy and Observe

> Now I come to the question of Client->TCP->Proxy->UDP->Server.  Since 
> the observe value is now to be ignored or removed, the first option 
> above will no longer work and I am not sure which the remaining 
> options is the correct one to do.  Does a reliable transmission 
> protocol mean that we should suppress items?

According to the data model that observe is based on, out-of-order observe
response packets are considered to be out-of-date and thus should be
ignored. Proxies must be aware of outstanding observations and keep track of
the observe counters to make sure that out-of-order packets are dropped. I'm
not super happy about the mechanism, but that seems to be what the standard
has settled on. Passing everything through and letting the client sort it
out is explicitly not allowed.

[JLS] I cannot quite see that as being a true statement.  It is true that
the most reason transmission must be considered the freshest, however a
client may want a more complete history of values than just the most recent
one and can deal with a non-freshest value.  For example, if it is a
controller and is doing projections based on the current set of values,
having the "stale" value is still useful information even if it is not the
freshest.  I do not see anything in RFC 7641 that says that out of order
packets are dropped by an intermediary.  Are you saying this just based on
the fact that it is not freshest?

Jim

-- RQ

> I have been sketching out what my TCP/UDP and UDP/TCP proxy code would 
> look like and I am having a problem deciding what should be done for 
> the observe option.
> 
> If you have the case of Client->UDP->Proxy->UDP->Server, then dealing 
> with out of order observations can be dealt with in the following ways:
> 
> 1. Pass them through unchanged or subtract a constant from the 
> observer value.  This pushes the problem of out of order observes onto the
client.
> 3. Keep out of order items around for a while and see if the out of 
> order items show up and transmit them back as in order.  I missing 
> items do not show up until the time out, then suppress the out of order
items.
> 4. Reorder the numbers so that things are seen in the wrong order by 
> the client.  (Obviously a wrong answer.)
> 
> Now I come to the question of Client->TCP->Proxy->UDP->Server.  Since 
> the observe value is now to be ignored or removed, the first option 
> above will no longer work and I am not sure which the remaining 
> options is the correct one to do.  Does a reliable transmission 
> protocol mean that we should suppress items?
> 
> Jim
> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Apr 2017 19:39:42 +0200

On Apr 24, 2017, at 19:26, Jim Schaad <[email-masked]> wrote:
> 
> I do not see anything in RFC 7641 that says that out of order

While there may be many kinds of proxy, the standard caching proxy will only put fresh data into the cache, and only those would lead to notifications.

Observing a resource may not be the best model if you want to keep an uninterrupted time series of the value of the resource.

Grüße, Carsten

---

From: Robert Quattlebaum <[email-masked]>
Date: Mon, 24 Apr 2017 10:54:07 -0700

Hello Jim,

> the fact that it is not freshest?

The dropping behavior is implied by two things:

* RFC7641 Section 5 says that proxies must track and renumber observe sessions
* draft-ietf-core-coap-tcp-tls-07 requires the observing endpoint to ignore the numeric value of the Observe header (because it is assumed to always be increasing).

I think the most reasonable way to reconcile these requirements is that an intermediary proxy MUST drop out-of-order observations. You could argue that such logic only applies when proxying to TLS/TCP/WebSockets, but I think the intent is for this behavior to be required independent of the raw transport.

I'm not saying that I like it or that I think that this is how it should have be, I'm just saying what the current documentation appears to require and what will likely be standardized.

-- RQ

---

From: Robert Quattlebaum <[email-masked]>
Date: Mon, 24 Apr 2017 10:57:32 -0700

Even a non-caching proxy will have to drop out-of-order observations if it proxies across UDP and TLS. There is pretty much only one correct behavior in that case, considering that there is no way to preserve the observe counter across a draft-ietf-core-coap-tcp-tls-07-style stream.

> On Apr 24, 2017, at 10:39 AM, Carsten Bormann <[email-masked]> wrote:
>