Subject: [core] Uri-Path and -Query segmentation

From: Zach Shelby <[email-masked]>
Date: Mon, 17 Oct 2011 13:37:33 +0300

Hi,

We have started gaining a good deal of implementation and interop experience around the coap-07 specification, and have also seen the first multi-vendor implementations of ETSI M2M with the CoAP binding. Overall the interoperability has been good, usually just a few issues to work out in the beginning. There is however one area of the specification I think we need to reconsider as a result of this experience - the segmentation of Path and Query URI components.

This segmentation is a neat trick that we felt would make constrained implementations easier and smaller. However so far I have not seen that benefit, while at the same time it is causing interoperability problems and does not save any overhead (replacing a delimiter character for an 8-bit option header, sometimes 16-bit!). One other problem that has come up, is that in some uses of URIs, the max number of CoAP options (15) quickly runs out. For example the ETSI M2M standard has some very deep (10 or so) resource structures, and may also have query parameters. 

I would like to propose that we back-track to the simpler solution we had in previous versions, just including the whole path or query string in the Uri-Path or -Query option, respectively. However, in order to overcome the 270 octet limit of a single option, we allow the concatenation of these options as is done in the Proxy-Uri option. This is a minor change for implementations, as many already just perform concatenation when processing these options. 

Thoughts?

This would require the following technical changes to coap-07:

Section 5.10.2:

-  o  each Uri-Path Option specifies one segment of the absolute path to
-     the resource, and
+  o  the Uri-Path Option contains the absolute path to the resource, and

-   o  each Uri-Query Option specifies one argument parameterizing the
-      resource.
+   o  the Uri-Query Option contains the query string of the resource.

Section 6.4:

   8.  If the value of the <path> component of /url/ is empty or
       consists of a single slash character (U+002F SOLIDUS "/"), then
       move to the next step.

 -     Otherwise, for each segment in the <path> component, include a
 -     Uri-Path Option and let that option's value be the segment (not
 -     including the delimiting slash characters) after converting all
 -     percent-encodings ("%" followed by two hexadecimal digits) to the
 -     corresponding characters.

+    Otherwise, the <path> component is included in a Uri-Path option 
+    after converting all percent-encodings ("%" followed by two hexadecimal 
+    digits) to the corresponding characters. In case the <path> doesn't fit 
+    within a single option, the Uri-Path Option MAY be included multiple 
+    times in a request such that the concatenation of the values results in 
+    the single <path>. All but the last instance of the Uri-Path Option MUST 
+    have a value with a length of 270 bytes, and the last instance MUST NOT be empty.

   9.  
-       If /url/ has a <query> component, then, for each argument in the
-       <query> component, include a Uri-Query Option and let that
-       option's value be the argument (not including the question mark
-       and the delimiting ampersand characters) after converting all
-       percent-encodings to the corresponding characters.

+      If /url/ has a <query> component, then the <query> component is included 
+      in a Uri-Query option after converting all percent-encodings to the corresponding 
+      characters. In case the <query> doesn't fit 
+      within a single option, the Uri-Query Option MAY be included multiple 
+      times in a request such that the concatenation of the values results in 
+      the single <query>. All but the last instance of the Uri-Query Option MUST 
+      have a value with a length of 270 bytes, and the last instance MUST NOT be empty.

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Olaf Bergmann <[email-masked]>
Date: Mon, 17 Oct 2011 14:28:25 +0200

Zach,

Zach Shelby <[email-masked]> writes:

> There is however one area of the specification I think we need to reconsider as a result of this experience - the segmentation of Path and Query URI components.

from my perspective, the segmentation and normalization of
percent-encodings significantly contribute to the handling of URIs on
the server-side as you can easily walk down on your sequence of path
segments without having to segment the URI yourself, and you can feed it
easily into a hash function to identify resources as it is already
normalized. 

[...]

On the other hand, for URI-Query, I do not see a big advantage of
splitting it, therefore, your proposal sounds good to me.  (The
combination of normalizing percent-encodings with the splitting into 270
bytes chunks is somewhat nasty where a multibyte character is split in
the middle -- this may make.implementation of client and server parsers
a bit more complex).

> +      have a value with a length of 270 bytes, and the last instance MUST NOT be empty.

Gruesse,
Olaf

---

From: "Rahman, Akbar" <[email-masked]>
Date: Mon, 17 Oct 2011 11:10:40 -0400

Hi Zach,

With regards to your suggestion:

	"I would like to propose that we back-track to the simpler
solution we had in previous versions, just including the 	whole
path or query string in the Uri-Path or -Query option, respectively.
However, in order to overcome the 270 	octet limit of a single option,
we allow the concatenation of these options as is done in the Proxy-Uri
option. 	This is a minor change for implementations, as many
already just perform concatenation when processing these
options."

I support this change (back to the simpler solution).  In fact, I was
considering the trade-off between complexity and benefit when I was
looking at this issue as part of the HTTP-CoAP Mapping draft, and I
agree with you that the current approach does not offer much benefit.

Akbar

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Zach Shelby
Sent: Monday, October 17, 2011 6:38 AM
To: core WG
Subject: [core] Uri-Path and -Query segmentation

Hi,

We have started gaining a good deal of implementation and interop
experience around the coap-07 specification, and have also seen the
first multi-vendor implementations of ETSI M2M with the CoAP binding.
Overall the interoperability has been good, usually just a few issues to
work out in the beginning. There is however one area of the
specification I think we need to reconsider as a result of this
experience - the segmentation of Path and Query URI components.

This segmentation is a neat trick that we felt would make constrained
implementations easier and smaller. However so far I have not seen that
benefit, while at the same time it is causing interoperability problems
and does not save any overhead (replacing a delimiter character for an
8-bit option header, sometimes 16-bit!). One other problem that has come
up, is that in some uses of URIs, the max number of CoAP options (15)
quickly runs out. For example the ETSI M2M standard has some very deep
(10 or so) resource structures, and may also have query parameters. 

I would like to propose that we back-track to the simpler solution we
had in previous versions, just including the whole path or query string
in the Uri-Path or -Query option, respectively. However, in order to
overcome the 270 octet limit of a single option, we allow the
concatenation of these options as is done in the Proxy-Uri option. This
is a minor change for implementations, as many already just perform
concatenation when processing these options. 

Thoughts?

Section 5.10.2:

-  o  each Uri-Path Option specifies one segment of the absolute path to
-     the resource, and
+  o  the Uri-Path Option contains the absolute path to the resource,
and

Section 6.4:

+    Otherwise, the <path> component is included in a Uri-Path option 
+    after converting all percent-encodings ("%" followed by two
hexadecimal 
+    digits) to the corresponding characters. In case the <path> doesn't
fit 
+    within a single option, the Uri-Path Option MAY be included
multiple 
+    times in a request such that the concatenation of the values
results in 
+    the single <path>. All but the last instance of the Uri-Path Option
MUST 
+    have a value with a length of 270 bytes, and the last instance MUST
NOT be empty.

   9.  

+      If /url/ has a <query> component, then the <query> component is
included 
+      in a Uri-Query option after converting all percent-encodings to
the corresponding 
+      characters. In case the <query> doesn't fit 
+      within a single option, the Uri-Query Option MAY be included
multiple 
+      times in a request such that the concatenation of the values
results in 
+      the single <query>. All but the last instance of the Uri-Query
Option MUST 
+      have a value with a length of 270 bytes, and the last instance
MUST NOT be empty.

Zach

---

From: Likepeng <[email-masked]>
Date: Tue, 18 Oct 2011 08:56:36 +0000

> I would like to propose that we back-track to the simpler solution we had in previous versions, just including the whole path or query string in the Uri-Path or -Query option, respectively.

I support this proposal. 

During our implementation, we found out that if multiple URI-Paths are out of order, it will introduce some problems. Also concatenation of URI-Path / URI-Query will introduce some problems.

So I think it is a good idea to have just one URI-Path or URI-Query. It will make easier for the implementation.

Kind Regards
Kepeng
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zach Shelby
Sent: Monday, October 17, 2011 6:38 PM
To: core WG
Subject: [core] Uri-Path and -Query segmentation

Hi,

Thoughts?

Section 5.10.2:

Section 6.4:

   9.  

Zach

---

From: "Dijk, Esko" <[email-masked]>
Date: Wed, 19 Oct 2011 10:00:47 +0200

Hi Zach,

Agree with the proposed change.

Esko

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zach Shelby
Sent: Monday 17 October 2011 12:38
To: core WG
Subject: [core] Uri-Path and -Query segmentation

Hi,

Thoughts?

Section 5.10.2:

Section 6.4:

   9.

Zach

The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: Zach Shelby <[email-masked]>
Date: Wed, 19 Oct 2011 11:37:28 +0300

Hi Olaf,

> easily into a hash function to identify resources as it is already

You bring up two good points.

1. Walking down the sequence of path segments is a nice feature, as long as you store your resources like that. What I have noticed is that most constrained devices have a small number of resources, so the whole path is just stored as a string (not individual segments). You then just do a string match or you can match a hash of the whole path. 

2. You are saying that normalization is somehow tied to segmentation of the path. In the proposed update normalization would be performed in exactly the same way as it is in -07, but instead of segmentation the whole normalized path is just placed in a Uri-Path option. Did I miss something here? Yes, you do have the same issue as pointed out below with Uri-Query, which means you need to concatenate before parsing.

> the middle -- this may make.implementation of client and server parsers

Yes, but the Uri-Query options would be first concatenated before parsing. The same goes for Uri-Path. In that case the multibyte characters don't cause a problem. 

I am not sure that we even really need to support path or query components larger than 270 bytes? It makes sense for Proxy-Uri when we are dealing with a potentially large absolute URI. 

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 31 Oct 2011 13:40:43 +0100

Zach,

coap-07 is fully consistent, as it sends fully parsed CoAP URLs only.
Apart from taking away any need for a CoAP server to do any parsing,
this particularly means coap-07 has full data transparency, e.g. you
could use a binary number as a path component.  For Uri-path, falling
back to coap-03 would take back that ability, as bytes of the value
0x2F are suddenly special again.  Similarly, Uri-query components
suddenly couldn't use 0x25 bytes any more.  There is no percent
encoding any more to work around this.

I believe support for binary values in URIs is important for CoAP.

The 15 option limit may indeed hurt more complex applications; maybe
we need to do something about that, but preferably not by futzing
around with every single repeatable option.

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 31 Oct 2011 16:36:43 +0200

> Apart from taking away any need for a CoAP server to do any parsing,

Yes, this has been pointed out clearly in the draft. I think it makes sense for servers that actually do parse the path into segments for comparison with its resources. What I have noticed is that most implementations seem to be happy comparing the whole path as many store it that way anyways. Parsing to find / is not much more difficult than parsing option headers, so although this is a nice feature, it is a marginal benefit only when parsing is actually needed. 

> could use a binary number as a path component.  

Ah, so that is the issue. I checked through the draft and we aren't really clear about this feature at all. It was only alluded at in one small piece of text that arbitrary information can be placed in a segment. 

> For Uri-path, falling
> 0x2F are suddenly special again.  

Hmm... how does that actually work? Section 6.4 prescribes these steps:

1. CoAP client gets request URI from upper layer application
2. As per 6.4 the request URI is split into CoAP options
3. In order to split the <path> component into segments the following step is used:

"Otherwise, for each segment in the <path> component, include a
       Uri-Path Option and let that option's value be the segment (not
       including the delimiting slash characters) after converting all
       percent-encodings ("%" followed by two hexadecimal digits) to the
       corresponding characters."

0x2F is special here as well, as any path that had arbitrary binary content would be split by 0x2F if it shows up in the octet sequence. Is the use of 0x2F inside of a path segment even legal as per RFC3986? All CoAP client APIs I have seen so far just accept the request URI path as an octet sequence, and they actually use the Section 6.4 process to produce Uri-Path options.

Although placing arbitrary binary could be made to work by not following section 6.4 and just manually placing that content into each Uri-Path option, this is not at all obvious from the draft that we have this feature (and why) or from Section 6. Also what does this mean on the server side? How does it know this is binary content and not a path segment? What happens when it combines the options back into a complete path?

> Similarly, Uri-query components

I think this needs some more discussion. 

> around with every single repeatable option.

From what I can tell we will run across this issue with REST designs that need to work on both HTTP and CoAP as in the ETSI M2M case. All you need is a 6-8 segment path, with 3-4 query parameters and some common CoAP options and you hit the limit. Of course if you make the REST design specifically to reduce path depth with CoAP in mind you can avoid that, but this is not really practical in all cases. 

Changing the 15 option limit would involve modifying that base CoAP header - I would really rather not do that. Or what did you have in mind?

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 31 Oct 2011 16:03:59 +0100

Zach,

I think a good intro to the usual use of percent-encoding is in

	http://en.wikipedia.org/wiki/Percent-encoding

This specifically addresses the use of %2F in paths, which by the way is rather common in the HTTP space as in

http://www.example.com/myniceproxy/http%3A%2F%2Fwww.ietf.org/

> 0x2F is special here as well, as any path that had arbitrary binary content would be split by 0x2F if it shows up in the octet sequence.

No, that's not what it says.  Percent decoding is after segment (what should be called "path component", i.e. the things between the slashes) splitting.

> Although placing arbitrary binary could be made to work by not following section 6.4 and just manually placing that content into each Uri-Path option, this is not at all obvious from the draft that we have this feature (and why) or from Section 6.

One of the problems of a spec like CoAP is that we inherit the world opened up by the referenced specs.
URIs have had the ability to encode binary data from the point where percent-encoding was introduced.
So the text of CoAP doesn't mention this, as it is not anything special.
What *is* special, is that these binary data are actually encoded in raw form in the protocol, and this point maybe needs to be spoken to explicitly.

> Also what does this mean on the server side? How does it know this is binary content and not a path segment?

There is no difference.  All path segments are sequences of (binary) bytes.
It is up to the server to structure its URI space, and if it does this by assigning semantics to the path segments that are based on binary arithmetic, more power to it.

> What happens when it combines the options back into a complete path?

It has to do the percent encoding to preserve those bytes that are not unreserved (section 2.3) in 3986.

I'm painfully aware that many HTTP users (read: many of my students :-) are not aware of the complexity of URI encoding.  This complexity is there, we can't reduce it, we just can try to isolate the CoAP protocol as much as possible from it.  I think that coap-07 succeeds rather well in this.  (What remains of the complexity is mostly in the HTTP mappers -- that's maybe why the authors of these perceive CoAP to be complex; when in reality it is HTTP and its URIs that are complex.)

> […]
> 6-8 segment path,

YMBK :-)

> Changing the 15 option limit would involve modifying that base CoAP header - I would really rather not do that. Or what did you have in mind?

See the upcoming coap-misc-09 for a strawman.  Not sure I like what I'm writing there, though.

Grüße, Carsten

---

From: "Kovatsch Matthias" <[email-masked]>
Date: Mon, 31 Oct 2011 18:19:31 +0000

Hi

What about introducing the general rule that when all bits of a length/count field are ones, another byte is following, whose value is added (so there might be multiple bytes following, allowing sizes > 270).
No memmove for concatenation would be required. Also, we would not have any length limitations, which almost always turned out bad for other systems in the past.
For the option count, we now saw that 15 might be too low. Even when URI-Path and -Query become singletons again, we might have something similar to the HTTP X-headers in the future. Hopefully, the unlikeliness of 15 options so far does not break compatibility with the current base CoAP header (Carsten, I could not find coap-misc-09 to see what is said in there...).
Maybe the additional OC bytes should come after the 4 bytes base header, as many might use the constant length 4 in their implementations...

I think length the limitations are bad if we want CoAP to be future proof.

Best regards
Matthias

---

From: Zach Shelby <[email-masked]>
Date: Mon, 31 Oct 2011 22:46:13 +0200

Carsten,

Thanks, that helped clarify the issue. Now even RPL doesn't seem all that complicated :-) I was confused by the term "binary data", and you meant percent-encoded data. I suggest we let this ticket rest for now, and do some more homework in Taipei. 

Just a few more questions on this for now: 

- It seems to me that UTF-8 is what is causing the need for splitting the segments. If the percent encoding would stay un-touched we would be fine, or does that move the normalization problem to the server? Of course percent-encoding is less efficient over the air than UTF-8.

- If "/" is the only character that is a problem in the Uri-Path, couldn't we just keep representing that as %2F? 

- As we are defining the coap(s):// scheme, couldn't we have done anything more to limit our exposure to this problem?

Zach

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 31 Oct 2011 23:22:13 +0100

On Oct 31, 2011, at 19:19, Kovatsch Matthias wrote:

> (Carsten, I could not find coap-misc-09 to see what is said in there...).

Fixed:

	http://tools.ietf.org/html/draft-bormann-coap-misc-09#section-2

Grüße, Carsten

---

From: "Kovatsch Matthias" <[email-masked]>
Date: Tue, 1 Nov 2011 01:28:25 +0000

> 	http://tools.ietf.org/html/draft-bormann-coap-misc-09#section-2

Thanks!

> See the upcoming coap-misc-09 for a strawman.  Not sure I like what I'm writing there, though.

I definitely like the opinion on the artificial limitations :)
I would, however, vote for a consistent rule, as described in my previous mail: "all ones -> additional length/count byte following" (with arbitrarily many additional length/count bytes). Also in preference over the more efficient 2-byte solution of Figure 5.

Best regards
Matthias

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 1 Nov 2011 06:47:19 +0100

> I definitely like the opinion on the artificial limitations :)

Indeed, this should now be common knowledge about system (and protocol) design.
The interesting question is where the "painless" threshold is.

> I would, however, vote for a consistent rule, as described in my previous mail: "all ones -> additional length/count byte following" (with arbitrarily many additional length/count bytes). Also in p…

Ah, you are a fan of unary encoding (base 255) :-)

Re the number of options, I now have yet another proposal (too late for the draft deadline):

2.1.4.  Alternative: Going to a delimiter model

   Another alternative is to spend the additional byte not as an
   extended count, but as an option terminator.

   In this proposal setting OC to 15 means that the number of options is
   no longer given explicitly.  Instead, the sequence of options is
   terminated by a byte with a special interpretation, 0xF0.  (Note that
   the option _delta_ of 15 is made special, not any specific option
   number.)  The next byte is then the first byte of the payload.

     0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Ver| T |  15   |      Code     |          Message ID           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |   Options ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |1 1 1 1 0 0 0 0|   Payload (if any) ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 5: Option Terminator for CoAP Messages with 15+ options

   (It is to be decided whether 0xF0 is made special only for OC=15,
   i.e. if it retains its meaning of (option delta = 15, option length =
   0) for other values of OC.)

Of the alternatives, I hate this one the least so far.

Grüße, Carsten