Subject: [core] #320: Nail down who does Unicode normalization when (or not)

From: "core issue tracker" <[email-masked]>
Date: Mon, 29 Apr 2013 16:17:51 -0000

#320: Nail down who does Unicode normalization when (or not)

 Ted Lemon notes (msg04287a):

 Section 3.2

 P. 19: I'm a little uncomfortable with this text:

 There is no expectation and no need to perform normalization within a CoAP
 implementation (except where Unicode strings that are not known to be
 normalized are imported from sources outside the CoAP protocol).

 I think what's intended here is right, but you've mentioned what amounts
 to a strong suggestion, if not a requirement, as a parenthetical note.  It
 seems like what you intended was something like this:

 There is no expectation and no need to perform normalization within a CoAP
 implementation, since senders are expected to be implemented with pre-
 normalized strings.  However, strings received from non-CoAP sources
 SHOULD be normalized where possible.

 Of course, there's actually no value to normalization in this case if it
 can't be depended on, and I suspect that you don't want to make that a
 MUST.   So this might be a better way to do it:

 There is no expectation and no need to perform normalization within a CoAP
 implementation, since senders are expected to be implemented with pre-
 normalized strings.  Strings received from non-CoAP sources and then
 forwarded by CoAP senders cannot be assumed to have been normalized, and
 may not compare correctly with normalized strings representing the same
 text.

 I don't have a strong opinion about how this should be done, but it seems
 like the text as written doesn't give clear guidance.  It seems that
 cross-proxies ought to be able to do normalization, and maybe other
 proxies could as well, but that's a much bigger change.

 Carsten says:

 Unicode normalization is always a sticky point.  I think the choice of a
 point on the continuum between your suggestions (which I both like) needs
 to be discussed with the WG.

-- 
-------------------------+-------------------------------------------------
 Reporter:               |      Owner:  [email-masked]
  [email-masked]           |     Status:  new
     Type:  other        |  Milestone:  post-IESG-1
  technical              |    Version:  coap-15
 Priority:  minor        |   Keywords:
Component:  coap         |
 Severity:  Submitted    |
  WG Document            |
-------------------------+-------------------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/core/trac/ticket/320>
core <http://tools.ietf.org/core/>

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 30 Apr 2013 23:11:20 +0200

So here is my proposal for the sentenced addressed by #320:

                     There is no expectation and no need to perform
                    normalization within a CoAP implementation, since senders
                    are expected to be implemented with pre-normalized
                    strings.  However, strings received from non-CoAP sources
                    need to be normalized when it is important that
                    they compare correctly with normalized strings
                    representing the same text.

(this is on page 19, in the text that explains what we mean by an option type "string".)

This changes the existing semi-prescriptive "not expected to normalize except on import" into a simple, true descriptive statement about what will go wrong if you don't.  I think this is exactly what we can do here.

I'll put this in tomorrow if people agree with this approach.

Grüße, Carsten