Subject: Re: [core] Comments on draft-ietf-core-link-format-04

From: Zach Shelby <[email-masked]>
Date: Wed, 1 Jun 2011 18:44:53 +0200

On Jun 1, 2011, at 5:46 AM, Cullen Jennings wrote:

> It's a binary string that is bit wise compared and is capable of caring a UTF-8 value. I guess this discussion is helping me get clear on explaining what I think implementors want. I think they want to treat these as binary strings and they are fine with requirement that these binary strings be capable of caring UTF-8 data but there not handling it as UTF-8 data. They are taking a more generic handling of it and don't want to do the things UTF-8 processing would require. Does this make any sense or sound reasonable?

Exactly. So we need to define "The CoRE link format MUST use UTF-8 encoding [RFC3629]" without mentioning NFC (as Alexey also recommended) with a paragraph about how we expect implementations to treat this. I will make a ticket to this extent and work on some text. 

Zach

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://www.sensinode.com
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: +358 40 7796297

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 04 Jun 2011 21:53:39 +0100

Cullen Jennings wrote:

>On May 26, 2011, at 11:14 AM, Alexey Melnikov wrote:
>  
>
>>Cullen Jennings wrote:
>>    
>>
>>>On May 24, 2011, at 3:03 PM, Alexey Melnikov wrote:
>>>      
>>>
>>>>Zach Shelby wrote:
>>>>        
>>>>
>>>>>On May 4, 2011, at 8:29 PM, Cullen Jennings wrote:
>>>>>          
>>>>>
>>[...]
>>    
>>
>>>>>>Related to this, where it says the CoRE link format MUST be in UTF-8 and SHOULD be NFC. This seems wrong unless we are updating RFC5988. RFC5988 makes it clear the links are as defined in rfc3986. We don't need to say any more or less than RFC5988. Unless someone can tell me how my implementation and interoperability changes by specifying UTF-8 and NFC, I'd like to see that removed. The phrase "MUST be compared as strings" is pretty vague when discussing UTF-8 strings. If people disagree here, I'm going to argue that unicode has a perfectly good symbol for angstrom and if someone wants to use that, I think they should be able to use it. Given this is format is for machine readability, an angstrom is not the same as a character that might look the same when printed on paper and that difference should not be lost.  All in all, the documents this depends on, such as 5988 deal with the right level of using UTF-8. This draft should just be silent on the i
>>>>>>ssue and you can probably remove all mention of UTF-8. Regardless of the UTF-8 issues, it seems the media type registration should have binary not 8bit.                    
>>>>>>
>>>>>This is how I started out, but in a couple review's a couple versions back it was suggested that we specify UTF-8 and we made a ticket for that. The intention is definitely to be in line with RFC5988, so I would be more than happy to have no mention of UTF-8 in the document and instead just say "the links are as defined in rfc3986". Again, will make a ticket if I hear no objections.
>>>>>          
>>>>>
>>>>But RFC 3986 only covers URIs and CoRE link format can include other data?
>>>>I think saying that the format contains UTF-8 is actually useful for future extensibility.
>>>>        
>>>>
>>>I think the part I am getting hung up on is what is allowed, and what needs to be checked and enforced by everyone. I think most the WG agrees they want implementation to treat things as just a bunch of bits that is passed around with no bitwise changes and are compared bit for bit.
>>>      
>>>
>>That is fine. But this doesn't mean that data is binary (for example).
>>    
>>
>>>They are perfectly happy if these bit strings turn out to be UTF-8 strings and no desire to limit it to US-ASCII for the stuff that is meant for humans instead of machines. I suspect you are roughly on the same page but the question is how to get the specification to say that clearly enough that implementors don't get confused. Some of this confusion is introduced by people that don't understand core and are arguing against it's use - for example I recently saw a draft presentation that claimed small constrained nodes implement core would need a full XML library and unicode string process library. I straightened out that presentation but was I like the specs to be clear enough that I did not see stuff like that. 
>>>Thoughts on how to get this clear in the main coap draft as well as here.
>>>      
>>>
>>I think at least saying that any CORE link format payload is UTF-8 is useful. This tells implementations what they should generate and what they can expect. And if they want to check for UTF-8 and reject invalid data, they can. If you want to say more than "MUST be UTF-8", then you should probably have separate statements about what compliant generators should produce and what consumers should do.
>>
>>NFC or any other form of Unicode normalization is useful for comparison or lookup of values. If UTF-8 data is only used for display, I don't think this is needed.
>>    
>>
>
>The stuff I am concerned about is not used for display to humans, but it is used for comparisons in machine to machine applications. We can ask the WG but the I think the odds are pretty good no one will want to implement NFC on a constrained devices.
>
I already proposed to drop the requirement to use NFC. I agree it is 
expensive to implement.

>As far as I can tell it's not UTF-8 as far as these devices are concerned. It's a binary string that is bit wise compared and is capable of caring a UTF-8 value. I guess this discussion is helping me get clear on explaining what I think implementors want. I think they want to treat these as binary strings and they are fine with requirement that these binary strings be capable of caring UTF-8 data but there not handling it as UTF-8 data. They are taking a more generic handling of it and don't want to do the things UTF-8 processing would require. Does this make any sense or sound reasonable?
>  
>
Each Unicode character has a single representation in UTF-8, so 
octet-by-octet comparisons work just fine for data in UTF-8.

My question is: should invalid UTF-8 (e.g. an octet 0xFF) be allowed?

If invalid UTF-8 should not be allowed, then some CORE agents can choose 
to verify UTF-8 (And others will choose not to. Maybe most of them). 
Verifying UTF-8 validity is really easy, especially when compared to 
implementing NFC.

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 6 Jun 2011 16:14:23 -0600

On Jun 4, 2011, at 2:53 PM, Alexey Melnikov wrote:

>>>> They are perfectly happy if these bit strings turn out to be UTF-8 strings and no desire to limit it to US-ASCII for the stuff that is meant for humans instead of machines. I suspect you are roughly on the same page but the question is how to get the specification to say that clearly enough that implementors don't get confused. Some of this confusion is introduced by people that don't understand core and are arguing against it's use - for example I recently saw a draft presentation that claimed small constrained nodes implement core would need a full XML library and unicode string process library. I straightened out that presentation but was I like the specs to be clear enough that I did not see stuff like that. Thoughts on how to get this clear in the main coap draft as well as here.
>>>>     
> I already proposed to drop the requirement to use NFC. I agree it is expensive to implement.
> 
> Each Unicode character has a single representation in UTF-8, so octet-by-octet comparisons work just fine for data in UTF-8.
> 
> If invalid UTF-8 should not be allowed, then some CORE agents can choose to verify UTF-8 (And others will choose not to. Maybe most of them). Verifying UTF-8 validity is really easy, especially when compared to implementing NFC.
> 
> 

Ok - sounds like wwe are the same page. I really don't care about if 0xFF is allowed or not. Zach's going to have a hard time writing the text to make this clear.