Subject: Re: [core] Block mechanism and size of ressources+query

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 2 Oct 2012 08:21:16 +0000

Hi Mathieu,

indeed a sort of BlockID Option could help to avoid the Uri + Uri-Query overhead and possibly overhead of other options too. Also, simple servers supporting -block, but that do not want to store such state, can simply not use the BlockID in their response.

Note with the current -block specification, there is already a proprietary way to avoid the overhead of Uri + Uri-query in each block request e.g. for Blockwise PUT:

*         Client sends a POST to the resource that it wants to update, including the long Uri-Path and Uri-Query options

*         Server responds with a single Location-Path option of a new temporary resource that was created e.g. a short random string "9X"
This resource is an alias to the actual resource that was addressed in the POST.

*         Client then PUTs / uploads blocks to the (short) resource path e.g. "9X", without Uri-Query options.

*         After full block transfer is done, the server removes the temporary resource

Optionally the client could indicate in the first step with a specific Uri-Query option that it want to apply the above technique (e.g. "create-alias=1" or so)

Esko

From: [email-masked] [mailto:[email-masked]] On Behalf Of M Pouillot
Sent: Monday 24 September 2012 17:11
To: [email-masked]
Subject: [core] Block mechanism and size of ressources+query

Hello all,

Is there a mean in the block mechanism to not send at each blocked packet all the ressource and query on which the request is done?
The size of the uri could be very big and so take a lot of bandwith, energy, ... on lossy network.

May be we can imagine on the first transaction of Block to have in response a BlockID-SequenceNumber on which all next transaction on this ID could be suffisant to remember all the context(ressource+query)...

Any suggestion?

Mathieu

________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 2 Oct 2012 10:48:24 +0200

> ·         Client sends a POST to the resource that it wants to update, including the long Uri-Path and Uri-Query options
> ·         Server responds with a single Location-Path option of a new temporary resource that was created e.g. a short random string “9X”
> ·         Client then PUTs / uploads blocks to the (short) resource path e.g. “9X”, without Uri-Query options.
> ·         After full block transfer is done, the server removes the temporary resource

Exactly, this is the way it can be done while not leaving the REST framework too much if the overhead really hurts.
Bonus: The client could even explicitly manage the state, and, DELETE the temporary resource when it decides that the upload needs to be aborted.
(Klaus discussed this a while ago, naming this state a "session".  http://www.ietf.org/mail-archive/web/core/current/msg02992.html
Right now, all this state is managed implicitly, which is probably the right thing for most use cases.)

A long time ago, we actually considered an option to give a temporary name to a resource within the normal request methods.
See appendix B.3 of coap-misc.
At the time, we decided we didn't need this, so it's in the "cemetery" part of coap-misc.

> The size of the uri could be very big and so take a lot of bandwith, energy, … on lossy network.

Right, the classical response to this is "then don't do it" :-)
Don't make big URIs for things that are used in many transfers.

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Tue, 2 Oct 2012 12:22:02 +0300

> At the time, we decided we didn't need this, so it's in the "cemetery" part of coap-misc.

There is nothing that stops you from doing this now, as multiple URIs can be associated with the same logical resource. The issue is just how you express that e.g. using the link format in order to discover alternative URIs for a resource (e.g. for the purpose on block transfer). For example RFC5988 points out an existing relation type called "alternate", not 100% sure if that applies here though. 

The alternative is of course to build something into the Block mechanism as Esko suggested. Just a thought, how hard would it be to use Token for that purpose? That is in practice what we use token for in Observe relationships. 

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 2 Oct 2012 13:30:09 +0000

The "alternate" relation could be used for shortening Uri-Path, however it can't cope with one or more Uri-Query Options in the blockwise request. These still have to be provided then in every request to PUT a block or request to GET a block of data.

Using a Token instead of a set of Uri-Path/Uri-Query seems possible in a block request following a previous one with the same Token, provided that the root resource ('/') is not assigned to something. This because a stateless server not supporting "sessions" via Token would interpret it as a request to get a block of the resource '/' , since Uri-Path Options are not included in the subsequent block request by the client.
The idea is then that a server not supporting the proposed use of Token would reply 4.04 Not Found, indicating to the client that establishing a "session" via Token this way is not supported.

Esko

-----Original Message-----
From: Zach Shelby [mailto:[email-masked]]
Sent: Tuesday 2 October 2012 11:22
To: Carsten Bormann
Cc: Dijk, Esko; [email-masked]
Subject: Re: [core] Block mechanism and size of ressources+query

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 2 Oct 2012 16:02:06 +0200

On Oct 2, 2012, at 15:30, "Dijk, Esko" <[email-masked]> wrote:

> proposed use of Token

Well, the whole point of ticket #210 was to get rid of such a second meaning for Token, because we found out it hurts interoperability when different implementations handle the overloading in a different way.
Instead of overloading Token with this semantics, it is much cleaner to have something like a special URI or a Teri.

I think we have a good basic design, I'm just still not convinced we need the function.
Tell me about the usecase...

Grüße, Carsten

---

From: "Dijk, Esko" <[email-masked]>
Date: Tue, 2 Oct 2012 14:13:40 +0000

For me the basic design is ok, with use of a 'special URI' or simply short URIs as a way to avoid the issue that Mathieu mentioned. I just wanted to confirm that overloading Token this way is possible. (Even though ticket #210 indicates we better not do this.)

Esko

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]]
Sent: Tuesday 2 October 2012 16:02
To: Dijk, Esko
Cc: Zach Shelby; [email-masked]
Subject: Re: [core] Block mechanism and size of ressources+query

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 2 Oct 2012 16:14:32 +0200

On Oct 2, 2012, at 16:02, Carsten Bormann <[email-masked]> wrote:

> good basic design

To summarize this outside the context of coap-misc B.3 (which didn't even talk about -block):

Any request on a URI can return an elective option, called Teri here.
(B.3 made the Teri a separate namespace from URIs, but that is kind of an orthogonal decision.)

In B.3, a Teri is just an alternative name for the resource, with a duration that is the length in time the server promises to remember that alternative name.
So this is entirely on the REST level, similar to the way a Location header works (but of course different).

People who have implemented Block1 using sessions/contexts might prefer the Teri to be a name for just that state; i.e. it would go away on a response with the more bit unset; the client might be able to delete the context/session, etc.
(Of course, when you have multiple names for something, you need to decide whether DELETE deletes the name or the thing.)

A Teri would always be elective, so you can run the protocol without alt-naming the resource/the context/session.
We would need to think about the proxy implications, but I would assume Teri to be hop-by-hop in semantics.

Grüße, Carsten

---

From: M Pouillot <[email-masked]>
Date: Tue, 2 Oct 2012 16:54:23 +0000

Hello Carsten,

For example during the registration on a RD, a sensor needs to post all its ressources. In my case it can be represented up to 10 blocks of 64 Bytes. For each block, we repeat the uri of the RD(not so big) and the option query of the RD (lifetime, host,...), 25 Bytes for 64 Bytes of data... => 250 extra Bytes for only one sensor :-( during the bootstrapping.

cordialement,

Mathieu

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de Dijk, Esko
Envoyé : mardi 2 octobre 2012 16:14
À : Carsten Bormann
Cc : [email-masked]
Objet : Re: [core] Block mechanism and size of ressources+query

Esko

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]]
Sent: Tuesday 2 October 2012 16:02
To: Dijk, Esko
Cc: Zach Shelby; [email-masked]
Subject: Re: [core] Block mechanism and size of ressources+query

Grüße, Carsten

_______________________________________________
core mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: Zach Shelby <[email-masked]>
Date: Tue, 2 Oct 2012 21:42:01 +0300

> For example during the registration on a RD, a sensor needs to post all its ressources. In my case it can be represented up to 10 blocks of 64 Bytes. For each block, we repeat the uri of the RD(not…

This is surely a good use case to optimize for as an example. But maybe not the best use of Block. Why not use 6LoWPAN fragmentation to handle this? This is what we do in our 6LoWPAN networks to avoid the needed for very small block sizes. 

Zach

> <POST-Watteco-OPCL-2a60.pcap>_______________________________________________
> core mailing list
> [email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 3 Oct 2012 00:44:30 +0200

On Oct 2, 2012, at 18:54, M Pouillot <[email-masked]> wrote:

> <POST-Watteco-OPCL-2a60.pcap>

Hi Mathieu,

thanks for the hard data on this.  I wish more people would send in pcaps when they argue a point!

In this case, the resource directory could do the POST/Location-Path game that we had been discussing.  
(First POST generates temporary URI to which a blockwise PUT can be made.)
1 Packet exchange extra, and a change in the resource directory resources.
Or the temporary URI could even be provided in the original /rdr/reg POST.  
I don't want to second-guess that protocol here, just point out some alternatives.

Also, I see that you are already fragmenting.
Adding in an IPHC context for 2002:db8::1 would save some 8 bytes per packet.
Going to 16-bit addresses might also help.
(I'll also use your pcap to get some additional GHC results... Stay tuned.)

But more generally speaking -- how often do these POSTs happen?
Is it worth optimizing out those ~ 20 % of one direction of the exchange?

Grüße, Carsten

---

From: "Dijk, Esko" <[email-masked]>
Date: Wed, 3 Oct 2012 08:04:54 +0000

FYI  For RD, another tactic is to do a first registration with only part of the resources.
Then, an update of the registration which includes a second part of the resources.
If needed, a third update etc. until all resources are in the RD.

This helps because an update does not need anymore the Uri-Query parameters, only a short Uri-Path. And it also avoids excessive fragmentation. And it allows similar to -block to overcome the 1280 bytes limit which link-format descriptions may quickly run into.

Esko

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]]
Sent: Wednesday 3 October 2012 0:45
To: M Pouillot
Cc: Dijk, Esko; [email-masked]
Subject: Re: [core] Block mechanism and size of ressources+query

Hi Mathieu,

Grüße, Carsten

---

From: M Pouillot <[email-masked]>
Date: Thu, 4 Oct 2012 06:58:21 +0000

Hello Zach,

6lowpan fragmentation is not acknowledged from peer to peer... i don't want send another time all packets just for one loss

mathieu 

-----Message d'origine-----
De : Zach Shelby [mailto:[email-masked]] 
Envoyé : mardi 2 octobre 2012 20:42
À : M Pouillot
Cc : Dijk, Esko; Carsten Bormann; [email-masked]
Objet : Re: [core] Block mechanism and size of ressources+query

Zach

> De : [email-masked] [mailto:[email-masked]] De la part 
> de Dijk, Esko Envoyé : mardi 2 octobre 2012 16:14 À : Carsten Bormann 
> Cc : [email-masked] Objet : Re: [core] Block mechanism and size of 
> ressources+query
> 
> For me the basic design is ok, with use of a 'special URI' or simply 
> short URIs as a way to avoid the issue that Mathieu mentioned. I just 
> wanted to confirm that overloading Token this way is possible. (Even 
> though ticket #210 indicates we better not do this.)
> 
> Esko
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 4 Oct 2012 09:37:57 +0200

Well, you already are fragmenting.  If you go from 64 to 128, your request blocks still fit in two fragments each, but you have halved your overhead.

> 6lowpan fragmentation is not acknowledged from peer to peer... i don't want send another time all packets just for one loss

I don't know who the peers are here, but you can (and should) have hop-by-hop ACKs for frames both carrying unfragmented packets and fragments.
(I actually see those ACKs in your pcap, so you seem to be using them.)
Do you have actual measurements that show a problem (with reasonable numbers of fragments per packet, like, 2)?

Grüße, Carsten

---

From: M Pouillot <[email-masked]>
Date: Thu, 4 Oct 2012 08:00:10 +0000

Hi Carsten,

inline

cordialement,

Mathieu

-----Message d'origine-----
De : Carsten Bormann [mailto:[email-masked]] 
Envoyé : mercredi 3 octobre 2012 00:44
À : M Pouillot
Cc : Dijk, Esko; [email-masked]
Objet : Re: [core] Block mechanism and size of ressources+query

Hi Mathieu,

> could be interesting. I don't know if the RD allows that... What is defined in RD, does it keep the query context of the last POST?

> aaaah! we can have more than one context, I need to read more precisely the RFC :-)   
> we need to work on that :-)

> in our case for the moment, just one time at the bootstrapping for each node on the network to one RD.  
> it is always interesting to optimise. In our case we run in solar power alimentation, so all little gain to not send some data is indispensable.

Grüße, Carsten