Subject: [core] Comments on draft-ietf-core-link-format-04

From: Cullen Jennings <[email-masked]>
Date: Wed, 4 May 2011 11:29:09 -0600

Mediums sized issue:

I'm confused by the Encoding considerations in link-format-04 which are listed as "8bit (UTF-8 (NFC))". I suspect this should just say "binary". If this is really NFC, then does it mean constrained nodes need to do the canonicalization? Most places in the draft it makes it clear that you processes the data as binary data.  Every  implementation I have seen treats this as binary data - even section 4.1 describes a bitwise identical comparison. We also have the issue that RFC 2045 says 

   "8bit data" refers to data that is all represented as relatively
   short lines with 998 octets or less between CRLF line separation

and

   "Binary data" refers to data where any sequence of octets whatsoever
   is allowed.

I think what we have here is binary data which seems fine. This is meant to be be machine readable and it's not like we have to enforce that no lines is longer than 1000 octets. Yes, many of the strings in the binary blob of data may happen to be UTF-8 strings that are even in Normalized Form C but unless we get more specific about when and where this gets normalized I don't see how specifying this is NFC data helps. In fact most, if not all, of this is from a far more restricted grammar than UTF-8 NFC. 

Related to this, where it says the CoRE link format MUST be in UTF-8 and SHOULD be NFC. This seems wrong unless we are updating RFC5988. RFC5988 makes it clear the links are as defined in rfc3986. We don't need to say any more or less than RFC5988. Unless someone can tell me how my implementation and interoperability changes by specifying UTF-8 and NFC, I'd like to see that removed. The phrase "MUST be compared as strings" is pretty vague when discussing UTF-8 strings. If people disagree here, I'm going to argue that unicode has a perfectly good symbol for angstrom and if someone wants to use that, I think they should be able to use it. Given this is format is for machine readability, an angstrom is not the same as a character that might look the same when printed on paper and that difference should not be lost.  

All in all, the documents this depends on, such as 5988 deal with the right level of using UTF-8. This draft should just be silent on the issue and you can probably remove all mention of UTF-8. Regardless of the UTF-8 issues, it seems the media type registration should have binary not 8bit. 

Minor Issues:

The ABNF you are using for integer allows leading zero. Not sure you want that if you don't want to deal with if 041 matches 41. 

Has an review request been sent to ietf-types? I could not find one. You could even put the data of the review request was sent in the draft with a note to the RFC editor to remove that before publication. That makes things easier for future reviews as this goes through LC and IESG. 

It's never 100% clear to me how to till out the templates but I suspect that the template should have an 

Author: Zach Shelby <your email>

and a 

Change controller: IETF CORE Working Group delegated from the IESG.

Don't worry - if we get that wrong, it will get fixed when IANA does their LC review. 

We should consider removing section 7.4 and this draft is ahead of coap. The coap draft can just register this type with no need to mention it here. 

Cullen <with my individual contributor hat on>

---

From: Zach Shelby <[email-masked]>
Date: Mon, 9 May 2011 15:01:17 +0300

Cullen,

Thanks for the review, comments in-line. Would like to have Alexey's opinion on these too.

> I think what we have here is binary data which seems fine. This is meant to be be machine readable and it's not like we have to enforce that no lines is longer than 1000 octets. Yes, many of the st…

The 8bit was suggested by Alexey. I would be fine with "Binary data", your point makes sense. I will make a ticket if I don't hear any objections. 

> All in all, the documents this depends on, such as 5988 deal with the right level of using UTF-8. This draft should just be silent on the issue and you can probably remove all mention of UTF-8. Reg…

This is how I started out, but in a couple review's a couple versions back it was suggested that we specify UTF-8 and we made a ticket for that. The intention is definitely to be in line with RFC5988, so I would be more than happy to have no mention of UTF-8 in the document and instead just say "the links are as defined in rfc3986". Again, will make a ticket if I hear no objections.

> The ABNF you are using for integer allows leading zero. Not sure you want that if you don't want to deal with if 041 matches 41. 

Yep, will fix that. 

> Has an review request been sent to ietf-types? I could not find one.

Not yet, I have an AP to do that.

> You could even put the data of the review request was sent in the draft with a note to the RFC editor to remove that before publication. That makes things easier for future reviews as this goes through LC and IESG. 
> 
> 

Thanks.

> We should consider removing section 7.4 and this draft is ahead of coap. The coap draft can just register this type with no need to mention it here. 

Sure, this makes sense in my opinion. 

Zach

> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 24 May 2011 22:03:19 +0100

>Cullen,
Hi,
Sorry for the slow response.

>The 8bit was suggested by Alexey. I would be fine with "Binary data", your point makes sense. I will make a ticket if I don't hear any objections. 
If your data can include NULs, or bare LFs or CRs, or consist of long 
lines, then "binary" is fine.

>This is how I started out, but in a couple review's a couple versions back it was suggested that we specify UTF-8 and we made a ticket for that. The intention is definitely to be in line with RFC598…
But RFC 3986 only covers URIs and CoRE link format can include other data?
I think saying that the format contains UTF-8 is actually useful for 
future extensibility.

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 26 May 2011 10:24:11 -0600

On May 24, 2011, at 3:03 PM, Alexey Melnikov wrote:

> If your data can include NULs, or bare LFs or CRs, or consist of long lines, then "binary" is fine.

It can definitely have long lines but I'm can't think of a place that allows for NUL or other weird control characters. Take for example the rt attribute. Right now it is defined as RFC 2616 style quoted-string which means I can not really put UTF-8 data in it yet this is a place where I may want to call something "Temperature at <insert persons name> house". If Core treated that as any character other than double quote being allowed, we would be in fine shape for allowing people to put UTF-8 there. 

> I think saying that the format contains UTF-8 is actually useful for future extensibility.
> 

I think the part I am getting hung up on is what is allowed, and what needs to be checked and enforced by everyone. I think most the WG agrees they want implementation to treat things as just a bunch of bits that is passed around with no bitwise changes and are compared bit for bit. They are perfectly happy if these bit strings turn out to be UTF-8 strings and no desire to limit it to US-ASCII for the stuff that is meant for humans instead of machines. I suspect you are roughly on the same page but the question is how to get the specification to say that clearly enough that implementors don't get confused. Some of this confusion is introduced by people that don't understand core and are arguing against it's use - for example I recently saw a draft presentation that claimed small constrained nodes implement core would need a full XML library and unicode string process library. I straightened out that presentation but was I like the specs to be clear enough that I did not see stuff like that. 

Thoughts on how to get this clear in the main coap draft as well as here.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 26 May 2011 18:14:57 +0100

>>>On May 4, 2011, at 8:29 PM, Cullen Jennings wrote:
 [...]

>I think the part I am getting hung up on is what is allowed, and what needs to be checked and enforced by everyone. I think most the WG agrees they want implementation to treat things as just a bunch of bits that is passed around with no bitwise changes and are compared bit for bit.
>
That is fine. But this doesn't mean that data is binary (for example).

>They are perfectly happy if these bit strings turn out to be UTF-8 strings and no desire to limit it to US-ASCII for the stuff that is meant for humans instead of machines. I suspect you are roughly on the same page but the question is how to get the specification to say that clearly enough that implementors don't get confused. Some of this confusion is introduced by people that don't understand core and are arguing against it's use - for example I recently saw a draft presentation that claimed small constrained nodes implement core would need a full XML library and unicode string process library. I straightened out that presentation but was I like the specs to be clear enough that I did not see 
>stuff like that. 
>
I think at least saying that any CORE link format payload is UTF-8 is 
useful. This tells implementations what they should generate and what 
they can expect. And if they want to check for UTF-8 and reject invalid 
data, they can. If you want to say more than "MUST be UTF-8", then you 
should probably have separate statements about what compliant generators 
should produce and what consumers should do.

NFC or any other form of Unicode normalization is useful for comparison 
or lookup of values. If UTF-8 data is only used for display, I don't 
think this is needed.

---

From: Zach Shelby <[email-masked]>
Date: Sat, 28 May 2011 16:24:40 +0300

(changed the subject line)

On May 26, 2011, at 8:14 PM, Alexey Melnikov wrote:

>> They are perfectly happy if these bit strings turn out to be UTF-8 strings and no desire to limit it to US-ASCII for the stuff that is meant for humans instead of machines. I suspect you are roughly on the same page but the question is how to get the specification to say that clearly enough that implementors don't get confused. Some of this confusion is introduced by people that don't understand core and are arguing against it's use - for example I recently saw a draft presentation that claimed small constrained nodes implement core would need a full XML library and unicode string process library. I straightened out that presentation but was I like the specs to be clear enough that I did not see stuff like that. 
> I think at least saying that any CORE link format payload is UTF-8 is useful. This tells implementations what they should generate and what they can expect. And if they want to check for UTF-8 and reject invalid data, they can. If you want to say more than "MUST be UTF-8", then you should probably have separate statements about what compliant generators should produce and what consumers should do.
> 
> NFC or any other form of Unicode normalization is useful for comparison or lookup of values. If UTF-8 data is only used for display, I don't think this is needed.

I agree with Cullen here that we should not be doing anything differently than RFC5988, where the link format is defined as we are only adding a few link-extentions for it. The RFC5988 format includes both URIs and quoted strings, and defines a special attribute title* for use with RFC5987 encoding of different character sets or languages in a human readable title. In RFC5988 title and title* are the only attributes meant for human display. None of the new CoRE attributes are meant for human display. 

The only difference is that the CoRE Link Format is carried as a payload, rather than as an HTTP Header. I propose that we just add a statement that the "CoRE Link Format MUST conform to the rules of the HTTP header [RFC2616] and [RFC5988]" . The title* attribute defined by RFC5988 then allows for the use of RFC5987 encoding when an international title is needed. 

Zach 

http://www.sensinode.com

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 28 May 2011 22:28:43 +0200

> ... when an international title is needed. 

I don't know what an international title is.

Outdoor Temperature?
Außenthermometer?
Blåbærsyltetøy?
야외 온도계?
بیرونی ترمامیٹر؟
室外温度计?
屋外温度計?

I don't think any of these is "international" (well, most of them work in two or three countries).

RFC5987 is solving two problems:
-- the legacy mistake of defining ISO-8859-1 encoding for HTTP headers.  We don't have that problem.
-- carrying around language tags.  That may be a remaining reason to do RFC 5987.

I still think that link-format could simply define the character encoding to be UTF-8 and outlaw the use of any character encoding other than UTF-8 even in in RFC5987 usage.
There is no point in allowing the old tower of babel for anything new.

While it is true that many kinds of processing of data from human writing systems ("characters") is hideously expensive, I don't think that the simple act of representing a title in a link-format representation entails any of that complexity.  Entering from a keyboard, displaying, yes.  Carrying it around, no.  We don't have to be afraid of increasing complexity by removing unnecessary choice and saying it always is UTF-8.

Gruesse, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 30 May 2011 18:26:07 +0200

> There is no point in allowing the old tower of babel for anything new.

Right, this was the original intention by adding the UTF-8 language to core-link-format. Seems that the application area folks liked that approach as well, in recollection it was Martin Thomson's suggestion originally (ticket #84). This also matches the UTF-8 defined for the text/plain MIME type registry entry in core-coap and for CoAP URI options.  See

http://trac.tools.ietf.org/wg/core/trac/ticket/84

> While it is true that many kinds of processing of data from human writing systems ("characters") is hideously expensive, I don't think that the simple act of representing a title in a link-format r…

The text we still had in link-format-04 was 

"The CoRE link format MUST use UTF-8 encoding [RFC3629], which SHOULD be in NFC (Unicode Normalization Form C) as per Section 3 of [RFC5198]."

So Carsten, what do you suggest the UTF-8 text should be so that it achieves the intention of carrying around UTF-8 without any added complexity, or the misunderstandings Cullen is afraid of for readers of the document. 

And Cullen, now I am not sure what your intention was of your review comment about UTF-8. Was your comment that:
a) Yes, this should be encoded as UTF-8 but don't say anything about that as it is already defined by RFC5988 via RFC3986. It turns out those don't say that, as RFC5988 depends on the HTTP header. I removed the UTF-8 text from link-format-05 based on this assumption. 
*or*
b) We should be using the HTTP header encoding as is assumed in RFC5988. If so we would need to define that as it isn't obvious. 

Zach

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 31 May 2011 11:26:28 +0100

>-- the legacy mistake of defining ISO-8859-1 encoding for HTTP headers.  We don't have that problem.
Agreed.

I would think of CORE link-format as a different serialization mechanism 
for Link values.

>There is no point in allowing the old tower of babel for anything new.
I tend to agree.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 31 May 2011 11:34:02 +0100

Zach Shelby wrote:

>Right, this was the original intention by adding the UTF-8 language to core-link-format. Seems that the application area folks liked that approach as well,
>
Obviously I am +1.

>in recollection it was Martin Thomson's suggestion originally (ticket #84). This also matches the UTF-8 defined for the text/plain MIME type registry entry in core-coap and for CoAP URI options.
>
As a side note: URIs never contain non-ASCII.

>See
As per my earlier message, I suggest dropping ", which SHOULD be in NFC 
(Unicode Normalization Form C) as per Section 3 of [RFC5198]".

---

From: "Stok, Peter van der" <[email-masked]>
Date: Tue, 31 May 2011 16:03:27 +0200

Dear all,

The uri parsing algorithm in section 6.3 of coap-6 seems to imply that a uri-host option MUST be filled in when the uri contains the literal name of a host.
I prefer a MAY, such that uri-host can remain empty when the host name can be resolved to an IP address. With a MUST lots of unnecessary characters are transported.

Peter van der Stok

The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

---

From: Linyi Tian <[email-masked]>
Date: Wed, 01 Jun 2011 09:18:15 +0800

Hi, Peter

I believe the section 6.3 is only talking about the URI parsing for
Proxy-URI by the proxy. The first sentence implies this as follows:
"The steps to parse a request's options from a string /url/ are as follows".

In this case, the host MUST be present in the URI regardingless whether it
is IP-Literal, IPV4 or reg-name. Then step 5 in section 6.3 is correct. The
URI-Host MUST be filled with reg-name if no IP address was present.

The purpose of section 6.3 may need to be clarified for better readability. 

Cheers,
Linyi

>>-----Original Message-----
>>From: [email-masked] [mailto:[email-masked]] On Behalf Of
Stok,
>>Peter van der
>>Sent: Tuesday, May 31, 2011 10:03 PM
>>To: core WG
>>Subject: [core] Uri-host option
>>
>>Dear all,
>>
>>The uri parsing algorithm in section 6.3 of coap-6 seems to imply that a
uri-host
>>option MUST be filled in when the uri contains the literal name of a host.
>>I prefer a MAY, such that uri-host can remain empty when the host name can
>>be resolved to an IP address. With a MUST lots of unnecessary characters
are
>>transported.
>>
>>Peter van der Stok
>>
>>The information contained in this message may be confidential and legally
>>protected under applicable law. The message is intended solely for the
>>addressee(s). If you are not the intended recipient, you are hereby
notified that
>>any use, forwarding, dissemination, or reproduction of this message is
strictly
>>prohibited and may be unlawful. If you are not the intended recipient,
please
>>contact the sender by return e-mail and destroy all copies of the original
>>message.

---

From: Linyi Tian <[email-masked]>
Date: Wed, 01 Jun 2011 10:53:59 +0800

>>>>While it is true that many kinds of processing of data from human
writing
>>systems ("characters") is hideously expensive, I don't think that the
simple act
>>of representing a title in a link-format representation entails any of
that
>>complexity.  Entering from a keyboard, displaying, yes.  Carrying it
around,
>>no.  We don't have to be afraid of increasing complexity by removing
>>unnecessary choice and saying it always is UTF-8.
>>>>
>>>
>>>"The CoRE link format MUST use UTF-8 encoding [RFC3629], which SHOULD
>>be in NFC (Unicode Normalization Form C) as per Section 3 of [RFC5198]."
>>>
>>>So Carsten, what do you suggest the UTF-8 text should be so that it
achieves
>>the intention of carrying around UTF-8 without any added complexity, or
the
>>misunderstandings Cullen is afraid of for readers of the document.
>>>
>>>

[Linyi Tian] Is UTF8 used by default? Is this behavior defined in other RFC?
If yes, it should be fine to drop this but better to have a reference to
that RFC.

Cheers,
Linyi

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 31 May 2011 21:46:51 -0600

On May 26, 2011, at 11:14 AM, Alexey Melnikov wrote:

>>>>> Related to this, where it says the CoRE link format MUST be in UTF-8 and SHOULD be NFC. This seems wrong unless we are updating RFC5988. RFC5988 makes it clear the links are as defined in rfc3986. We don't need to say any more or less than RFC5988. Unless someone can tell me how my implementation and interoperability changes by specifying UTF-8 and NFC, I'd like to see that removed. The phrase "MUST be compared as strings" is pretty vague when discussing UTF-8 strings. If people disagree here, I'm going to argue that unicode has a perfectly good symbol for angstrom and if someone wants to use that, I think they should be able to use it. Given this is format is for machine readability, an angstrom is not the same as a character that might look the same when printed on paper and that difference should not be lost.  All in all, the documents this depends on, such as 5988 deal with the right level of using UTF-8. This draft should just be silent on the issue and you can probably remove all mention of UTF-8. Regardless of the UTF-8 issues, it seems the media type registration should have binary not 8bit.        

The stuff I am concerned about is not used for display to humans, but it is used for comparisons in machine to machine applications. We can ask the WG but the I think the odds are pretty good no one will want to implement NFC on a constrained devices. As far as I can tell it's not UTF-8 as far as these devices are concerned. It's a binary string that is bit wise compared and is capable of caring a UTF-8 value. I guess this discussion is helping me get clear on explaining what I think implementors want. I think they want to treat these as binary strings and they are fine with requirement that these binary strings be capable of caring UTF-8 data but there not handling it as UTF-8 data. They are taking a more generic handling of it and don't want to do the things UTF-8 processing would require. Does this make any sense or sound reasonable?