Subject: [Json] Character interop, “Noncharacters”?

From: Tim Bray <[email-masked]>
Date: Thu, 22 Aug 2013 10:45:47 -0700

I think we have consensus that naked surrogates are an area we’re going to
highlight as potentially leading to interop problems.

I’m wondering if Unicode “Noncharacters” fall into the same bucket.

=================================
Quoting from Unicode 6.2 section 2.4:

Noncharacters. Sixty-six code points are not used to encode characters.
Noncharacters
consist of U+FDD0..U+FDEF and any code point ending in the value FFFE16 or
FFFF16—
that is, U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, ... U+10FFFE, U+10FFFF. (See
Section 16.7, Noncharacters.)

And 16.7 begins:

Noncharacters are code points that are permanently reserved in the Unicode
Standard for internal use. They are forbidden for use in open interchange
of Unicode text data. See Section 3.4, Characters and Encoding, for the
formal definition of noncharacters and conformance requirements related to
their use.

====================================

So, does anyone think there’s any likelihood that generic
unicode-processing software is apt to barf or otherwise act in surprising
ways if one of these things turns up?  Worth calling out in -bis?

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 22 Aug 2013 20:04:04 +0200

On Aug 22, 2013, at 19:45, Tim Bray <[email-masked]> wrote:

> So, does anyone think there’s any likelihood that generic unicode-processing software is apt to barf or otherwise act in surprising ways if one of these things turns up?  Worth calling out in -bis?

Given the number of JSON implementations, you probably can find an instance for any unreasonable idea you can come up with.

https://github.com/couchdeveloper/JPJson/issues/4

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Thu, 22 Aug 2013 15:25:30 -0400

Tim Bray scripsit:

> I think we have consensus that naked surrogates are an area we’re
> going to highlight as potentially leading to interop problems.
>

[snip]

> And 16.7 begins:
>
> Noncharacters are code points that are permanently reserved in the
> Unicode Standard for internal use. They are forbidden for use in open
> interchange of Unicode text data. See Section 3.4, Characters and
> Encoding, for the formal definition of noncharacters and conformance
> requirements related to their use.

That part of Unicode has been updated by Corrigendum #9:  Clarification
About Noncharacters <http://www.unicode.org/versions/corrigendum9.html>,
which is retroactive to Unicode 3.1 and says in part:

	Noncharacters in the Unicode Standard are intended for internal
	use and have no standard interpretation when exchanged outside
	the context of internal use. However, they are not illegal in
	interchange nor do they cause ill-formed Unicode text. This
	has always been the intent of the standard, as expressed by the
	Unicode Technical Committee. This is necessary for the effective
	use of noncharacters, because anytime a Unicode string crosses an
	API boundary, it is in effect being "interchanged". Furthermore,
	for distributed software, it is often very difficult to determine
	what constitutes an "internal" versus an "external" context for
	any particular software process. The real intent of noncharacters
	is that they are permanently prohibited from being assigned
	standard, interchangeable meanings, rather than that they are
	prohibited from occurring in Unicode strings which happen to
	be interchanged.

> unicode-processing software is apt to barf or otherwise act in
> surprising ways if one of these things turns up?  Worth calling out in
> -bis?

My reading of the above is that it would have been all right to prohibit
noncharacters in JSON, but since that didn't happen, they should not be
considered problematic.  I know of no APIs that treat them specially.
So I would say no to both questions.

A fortiori, there seems no reason to expect problems with any other
class of Unicode code points.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
   There was an old man                Said with a laugh, "I
     From Peru, whose lim'ricks all      Cut them in half, the pay is
       Look'd like haiku.  He              Much better for two."
                                             --Emmet O'Brien

---

From: Nico Williams <[email-masked]>
Date: Thu, 22 Aug 2013 14:34:39 -0500

> class of Unicode code points.

+1.

IMO naked surrogates are the only Unicode problem for us to consider.
We could and should give advice as to what to do when receiving JSON
string values with naked surrogates, perhaps: "parsers SHOULD remove
[or replace with...] naked surrogates" and/or "parsers SHOULD signal
the presence of naked surrogates", or something along those lines.

---

From: John Cowan <[email-masked]>
Date: Thu, 22 Aug 2013 21:58:47 -0400

Carsten Bormann scripsit:

> Given the number of JSON implementations, you probably can find an
> instance for any unreasonable idea you can come up with.
> 

I've just added a comment to that page pointing to Corrigendum #9.

-- 
As you read this, I don't want you to feel      John Cowan
sorry for me, because, I believe everyone       [email-masked]
will die someday.                               http://www.ccil.org/~cowan
        --From a Nigerian-type scam spam

---

From: John Cowan <[email-masked]>
Date: Fri, 23 Aug 2013 01:16:12 -0400

> the presence of naked surrogates", or something along those lines.

I am absolutely opposed to that.  A JSON parser in an environment
where strings are naturally 16-bit code units (JVM, CLR, JavaScript)
can and most likely will treat unpaired surrogates as nothing
special.  (I'm assuming here that "naked" means "unpaired"
rather than "unescaped"; unescaped unpaired surrogates are
invalid encoding anyway, and below the level of JSON.)

-- 
Why are well-meaning Westerners so concerned that   John Cowan
the opening of a Colonel Sanders in Beijing means   [email-masked]
the end of Chinese culture? [...]  We have had      http://www.ccil.org/~cowan
Chinese restaurants in America for over a century,
and it hasn't made us Chinese.  On the contrary,
we obliged the Chinese to invent chop suey.            --Marshall Sahlins

---

From: Nico Williams <[email-masked]>
Date: Fri, 23 Aug 2013 01:21:45 -0500

> invalid encoding anyway, and below the level of JSON.)

You're right, and that was a brainfart on my part.  I'd forgotten that
my own preference had been to pass unpaired surrogates in escaped
form.  Please excuse the noise.