Subject: [Json] Unpaired surrogates in JSON strings

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 12:22:46 -0400

RFC 4627 Section 1 says:

    A string is a sequence of zero or more Unicode characters.

However, the grammar of strings permits things like "foo\uDC00bar",
which contains an escape sequence that does not correspond to any
Unicode character.  This provides backward compatibility with JavaScript,
where a string is not a sequence of characters but a sequence of UTF-16
code units.

If Section 1 is normative, then there is a contradiction with Section 4,
which says:

    A JSON parser MUST accept all texts that conform to the JSON
    grammar.

In my view, JSONbis processors should be REQUIRED to produce only strings
that conform to Section 1.

-- 
And it was said that ever after, if any                 John Cowan
man looked in that Stone, unless he had a               [email-masked]
great strength of will to turn it to other              http://ccil.org/~cowan
purpose, he saw only two aged hands withering
in flame.   --"The Pyre of Denethor"

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 10:46:48 -0700

> that conform to Section 1.
Such a requirement will be breaking. Breaking changes are out of scope.

I like the suggestion that section 1 should be talking about code points 
instead of characters.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 11:01:41 -0700

<definitely no hat>

> Such a requirement will be breaking. Breaking changes are out of scope.

How is that "breaking"? Section 1 has a definition of strings, and Section 4 says that the parser must accept all texts that conform to the grammar. Surrogate code points are not characters, according to the Unicode spec.

> I like the suggestion that section 1 should be talking about code points instead of characters.

That seems like a significant change that would cause parsers that currently follow Section 1 to fail.

--Paul Hoffman

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 11:19:53 -0700

> That seems like a significant change that would cause parsers that currently follow Section 1 to fail.
It is not a change, it is a clarification. JavaScript, Java, and many 
other languages have strings of code points, owing to their being set 
before Unicode grew surrogate pairs. So strings in those languages are 
composed of code points. JSON is tolerant of that reality.

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 14:47:02 -0400

> Such a requirement will be breaking. Breaking changes are out of scope.

How is it a breaking change to limit what documents are allowed to be
*generated*?

-- 
I don't know half of you half as well           John Cowan
as I should like, and I like less than half     [email-masked]
of you half as well as you deserve.             http://www.ccil.org/~cowan
        --Bilbo

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 11:57:13 -0700

> How is it a breaking change to limit what documents are allowed to be
Because JSON is currently being used in applications that deliver those 
codepoints.
I believe that ECMA cannot accept if this is changed.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 5 Jun 2013 23:35:17 +0000

I don't mind allowing surrogate pairs in \u notation.  But I think we
should specify what happens when you send just half of the pair.  For
example, adding this after section 2.5, graph 4:

Escape sequences between \uD800 and \uDFFF SHOULD be generated only as
valid UTF16 surrogate pairs (this SHOULD is only to allow backward
compatibility).  When encountering an invalid surrogate pair (such as
"foo\uD834bar" or "\uDD1E\uD834"), parsers MAY either throw an error
(taking the risk of some backward incompatibility with old generators) or
MAY ignore the sequence.

On 6/5/13 12:57 PM, "Douglas Crockford" <[email-masked]> wrote:

>_______________________________________________
>json mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/json
>

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 16:55:58 -0700

On Jun 5, 2013, at 11:57 AM, Douglas Crockford <[email-masked]> wrote:

> Because JSON is currently being used in applications that deliver those codepoints.

Can you say why an application would do that, given the JSON specification?

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 20:22:54 -0400

> Can you say why an application would do that, given the JSON specification?

Perhaps because its designers believe that Section 4's explicit mustard
trumps section 1's definition.

-- 
Her he asked if O'Hare Doctor tidings sent from far     John Cowan
coast and she with grameful sigh him answered that      http://ccil.org/~cowan
O'Hare Doctor in heaven was. Sad was the man that word  [email-masked]
to hear that him so heavied in bowels ruthful.  All
she there told him, ruing death for friend so young,    James Joyce, Ulysses
algate sore unwilling God's rightwiseness to withsay.   "Oxen of the Sun"

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 17:18:50 -0700

In section 2.5 of 4627, a reasonable reading of the text clearly disallows
unpaired surrogates, because the discussion is exclusively of characters,
which surrogates aren’t; they are code points, but there are no characters
that have those code points. From the introduction: “A string is a sequence
of zero or more Unicode characters”. Case closed.

A loose reading of the BNF probably allows naked surrogates if you ignore
what the text says.

I think anyone who’s delivering those codepoints is already in violation of
4627, and I don’t think we should retroactively forgive those sins.

-T

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 17:58:33 -0700

<no hat>

The text says "character" everywhere. It says "code point" exactly once, when it is talking about a character's code point.

The ABNF seems to allow code points, not characters. This is pretty clearly a mistake.

This could be resolved by a warning similar to what Joe proposed, but it should be stronger. Also, Joe's wording assumes that these code points are always represented as escape sequences, but the ABNF allows them to be in a string as raw, unescaped code points.

> (taking the risk of some backward incompatibility with old generators) or

Alternate proposal:

Code points between U+D800 and U+DFFF SHOULD be generated only as
valid UTF16 surrogate pairs; this SHOULD is only to allow backward
compatibility with applications that ignored the restriction that
strings consist of Unicode characters. A parser that encounters
an invalid surrogate pair (such as "foo\uD834bar" or "\uDD1E\uD834"),
SHOULD throw an error because the string does not consist of characters;
it might ignore the errant code points, but at the risk of allowing
strings that other parsers would find illegal.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 18:05:25 -0700

On Wed, Jun 5, 2013 at 5:58 PM, Paul Hoffman <[email-masked]> wrote:

> > valid UTF16 surrogate pairs
>

Turn it into MUST. The intent of 4627 is perfectly clear, even if the BNF
is buggy.  I don’t think this rewrite should allow things that the previous
spec didn’t. -T

> (this SHOULD is only to allow backward

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 18:08:23 -0700

On 6/5/2013 5:18 PM, Tim Bray wrote:
> In section 2.5 of 4627, a reasonable reading of the text clearly 
> disallows unpaired surrogates, because the discussion is exclusively 
> of characters, which surrogates aren’t; they are code points, but 
> there are no characters that have those code points. From the 
> introduction: “A string is a sequence of zero or more Unicode 
> characters”. Case closed.
>
> A loose reading of the BNF probably allows naked surrogates if you 
> ignore what the text says.
>
> I think anyone who’s delivering those codepoints is already in 
> violation of 4627, and I don’t think we should retroactively forgive 
> those sins.
>
> -T
>
>
> On Wed, Jun 5, 2013 at 4:55 PM, Paul Hoffman <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On Jun 5, 2013, at 11:57 AM, Douglas Crockford
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
The application that does that is JavaScript. Any 16-bit value can be 
put next to any other 16-bit value and then JSON encoded. The meaning of 
'character' throughout the RFC is ECMAScript's, which is roughly the 
same as Unicode's code point. This can be seen in the BNF. When the RFC 
talks about Unicode Characters, it is in the sense of Unicode 
Characters  and not EBCDIC Characters.

JSON is just the pipe. It doesn't need to be enforcing Unicode over 
JavaScript. The sender and receiver can argue about what it means to be 
a character. JSON has always been agnostic about this.

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 21:09:45 -0400

Tim Bray scripsit:

> of characters, which surrogates aren’t; they are code points,
> but there are no characters that have those code points. From the

Unfortunately it isn't.

> ignore what the text says.

It's not about a loose reading.  Section 4 says "A JSON parser MUST
accept all texts that conform to the JSON grammar".  That's a flat
contradiction of the above.

> violation of 4627, and I don’t think we should retroactively forgive

It's already been stated that ECMA can't swallow this change.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
Objective consideration of contemporary phenomena compel the conclusion
that optimum or inadequate performance in the trend of competitive
activities exhibits no tendency to be commensurate with innate capacity,
but that a considerable element of the unpredictable must invariably be
taken into account. --Ecclesiastes 9:11, Orwell/Brown version

---

From: John Cowan <[email-masked]>
Date: Wed, 5 Jun 2013 23:58:51 -0400

Douglas Crockford scripsit:

> When the RFC talks about Unicode Characters, it is in the sense
> of Unicode Characters  and not EBCDIC Characters.

That's just confusing.  "Unicode character" means something quite
definite in Unicode.  If you mean "character in the Unicode repertoire",
that's fine -- but all EBCDIC characters, in all variants of EBCDIC,
are just a subset of that.  There seems to me no reason why JSON
couldn't be encoded in EBCDIC, with appropriate escapes.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
In computer science, we stand on each other's feet.
        --Brian K. Reid

---

From: R S <[email-masked]>
Date: Wed, 5 Jun 2013 21:06:16 -0700

On Wed, Jun 5, 2013 at 8:58 PM, John Cowan <[email-masked]> wrote:

> That's just confusing.

Maybe, but I think we want to specify whatever it is browsers do with JSON
in this case. That amounts to billions of daily active C++ client users.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Jun 2013 06:17:27 +0200

On Jun 6, 2013, at 02:58, Paul Hoffman <[email-masked]> wrote:

> Code points

That is a confusing term here, and I have a hard time understanding what people are trying to say.

Code points can refer to those of the characters or those of the code units (byte for UTF-8, etc.).
Please be specific which of the ones you mean.

RFC 4627 normatively references RFC 4234, which makes very clear that it is about characters.
This is further reinforced by the grammar in RFC 4627, which addresses code points up to %x10FFFF -- clearly it is about the Unicode characters (an assessment that also agrees with the rest of the text).

1) Surrogates are not characters, so they can't appear in the source character set.

2) Within strings, escape sequences can be used to construct surrogates from ASCII characters.  This contradicts the definition of strings in section 1.  However, the usage of this escape sequences to represent arbitrary binary data appears to be common enough (JavaScript only recently grew support for binary data, and this support will not be retrofit into JSON) that a clarification in the specification is required.

Please separate these two items clearly.  Thanks.

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 00:29:21 -0400

> units (byte for UTF-8, etc.).

Code points are (mathematical) integers corresponding to Unicode
characters, though not all of them are assigned to characters.

Code *units* are 8-bit, 16-bit, or 32-bit values corresponding to the
components of a particular encoding.  A single code point is represented
by one or more code units.

-- 
Go, and never darken my towels again!           John Cowan
        --Rufus T. Firefly                      http://ccil.org/~cowan

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 21:48:35 -0700

> It's already been stated that ECMA can't swallow this change.

I thought ECMA’s indigestion was over dupe keys.

It seems to me that if unpaired surrogates are to be allowed, it’s not OK
for the spec to assert that strings are made of Unicode characters, because
both of these things can’t be correct.  -T

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 01:11:43 -0400

> both of these things can’t be correct.  -T

The author has said that "Unicode character" doesn't mean what the
rest of us thinks it means, though no definition I can understand has
been forthcoming.  However, standards (like other kinds of legal codes)
are basically interpreted by their users, not their authors: what the
author says has a peculiar interest, but not a peculiar authority.

-- 
John Cowan      <[email-masked]>       http://www.ccil.org/~cowan
                Charles li reis, nostre emperesdre magnes,
                Set anz totz pleinz ad ested in Espagnes.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Jun 2013 08:41:07 +0200

On Jun 6, 2013, at 03:08, Douglas Crockford <[email-masked]> wrote:

> JSON is just the pipe. It doesn't need to be enforcing Unicode over JavaScript. The sender and receiver can argue about what it means to be a character. JSON has always been agnostic about this.

JSON-the-format is about sequences of characters and can and probably should be character-encoding-scheme agnostic (as long as that character encoding scheme represents Unicode characters, because that's what the ABNF is about).

JSON-the-media-type (application/json) can't.  Media types describe the semantics of a sequence of bytes.
No characters in sight without defining how to get there.  So either we stick with auto-detecting one of the seven UTFs (with some implied or explicit semantics about BOMs, which is incompletely specified in RFC 4627) or we describe reality (UTF-8).

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 06:59:12 +0000

>characters, though not all of them are assigned to characters.

The intro to the Unicode standard makes this pretty clear:

http://www.unicode.org/versions/Unicode6.2.0/ch01.pdf

This is why I wanted to decouple from a particular version of Unicode.  If
the reference remained at version 4, for example, the word "character"
means that any code point not in that version of Unicode is not
technically legal JSON (although we know it will interop just fine in
practice, which is why it's pretty safe to do the update).

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 03:03:07 -0400

> practice, which is why it's pretty safe to do the update).

+1

-- 
One art / There is                      John Cowan <[email-masked]>
No less / No more                       http://www.ccil.org/~cowan
All things / To do
With sparks / Galore                     --Douglas Hofstadter

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 07:04:52 +0000

On 6/5/13 6:58 PM, "Paul Hoffman" <[email-masked]> wrote:

>This could be resolved by a warning similar to what Joe proposed, but it
>should be stronger. Also, Joe's wording assumes that these code points
>are always represented as escape sequences, but the ABNF allows them to
>be in a string as raw, unescaped code points.

When in non-encoded land, none of the half surrogate pairs will ever be
legal characters.  Those code points are forever reserved -- even though
many wish UTF16 would die a fire.  Now if we're talking about characters
with code points outside the BMP, then I agree.  The style guide could
suggest which of the several ways to encode them is best.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 07:16:56 +0000

On 6/5/13 7:08 PM, "Douglas Crockford" <[email-masked]> wrote:

>put next to any other 16-bit value and then JSON encoded.

Agree.  This is a relatively-outdated worldview that we're stuck with, and
is why I didn't say MUST in my suggested language.

>The meaning of 
>same as Unicode's code point.

I'm not convinced yet.

"\uD834\uDD1E".charCodeAt(0).toString(16);

Yields:

'd834'

That's not a code point.  That's half a surrogate pair for a code point
encoded in UTF16.  It's only the same in the BMP.

-- 
Joe Hildebrand

---

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 03:57:20 -0700

> couldn't be encoded in EBCDIC, with appropriate escapes.
Of course it is confusing. The text must be clarified to talk about 
codepoints.

---

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 04:03:28 -0700

On 6/5/2013 9:48 PM, Tim Bray wrote:
> On Wed, Jun 5, 2013 at 6:09 PM, John Cowan <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> It seems to me that if unpaired surrogates are to be allowed, it’s not 
> OK for the spec to assert that strings are made of Unicode characters, 
> because both of these things can’t be correct.  -T
I agree. The critical thing that Unicode contributes to JSON is the 
numbers used in \u forms. It is unfortunate that ECMAScript and Unicode 
use the word character so differently. So I think JSON should talk 
exclusively about Unicode Codepoints, because that provides the least 
ambiguity, and not use the word characters at all.

---

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 04:15:04 -0700

> encoded in UTF16.  It's only the same in the BMP.
What  then is the standard name for a 16-bit element of text? When 
JavaScript was created, that word was character. What is the word now?

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 08:23:34 -0400

> Of course it is confusing. The text must be clarified to talk about

Code *units*, specifically 16-bit ones, not code points (which are
mathematical integers).

-- 
And through this revolting graveyard of the universe the muffled, maddening
beating of drums, and thin, monotonous whine of blasphemous flutes from
inconceivable, unlighted chambers beyond Time; the detestable pounding
and piping whereunto dance slowly, awkwardly, and absurdly the gigantic
tenebrous ultimate gods --the blind, voiceless, mindless gargoyles whose soul
is Nyarlathotep. (Lovecraft)   John Cowan  [email-masked]

---

From: Tim Bray <[email-masked]>
Date: Thu, 6 Jun 2013 07:57:47 -0700

F0, 90, 8D, 86

> JavaScript was created, that word was character. What is the word now?

The only somewhat-standardized term would be “UTF-16 codepoint”.  But
that’s not really a “unit of text” any more than the 2nd byte of a
character encoded in 3 bytes with UTF-8 is.

I’m fairly shocked.  I have always believed that JSON encodes what its
introduction (and section 2.5 "Strings") say it encodes, Unicode
characters.

If it is a requirement to accommodate the class of bug where languages that
use UTF-16 (Java, JavaScript, C#) can emit unpaired UTF-16 surrogates, the
spec needs to be clear that the INTENT is actually to support Unicode
characters, and that unpaired surrogates are always evidence of a bug, and
there can be no expectation that any software receiving such buggy data
will be able to do anything useful with it, or even avoid crashing in a
hard-to-debug way down in the bowels of a library routine.  -T

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 11:08:29 -0400

+1

"If Parliament does not mean what it says, it must say so."

-- 
[W]hen I wrote it I was more than a little              John Cowan
febrile with foodpoisoning from an antique carrot       [email-masked]
that I foolishly ate out of an illjudged faith          http://ccil.org/~cowan
in the benignancy of vegetables.  --And Rosta

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 08:27:39 -0700

> practice, which is why it's pretty safe to do the update).

This is an *extremely* good point. The definition of "character" changes from version to version of Unicode, and it is now clear we need to deal with that in this document.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 11:33:01 -0400

Paul Hoffman scripsit:

> This is an *extremely* good point. The definition of "character"

Rather, the enumeration of characters.

> changes from version to version of Unicode, and it is now clear we
> need to deal with that in this document.

Do you mean "not clear"?

-- 
All Norstrilians knew that humor was            John Cowan
"pleasurable corrigible malfunction".          [email-masked]
        --Cordwainer Smith, Norstrilia

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 17:32:22 +0000

+1

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 17:49:04 +0000

>JavaScript was created, that word was character. What is the word now?

Let's say rather that we understand Unicode better than we did then.  When
we only cared about the Basic Multilingual Plane, it was easy to just say
"wchar_t", and hope the compiler writer knew more than we did.
Technically, wchar_t was supposed to be big enough to hold any supported
code point, but in practice, it was usually 16 bits.  This forced people
to start using UTF16 in wchar_t's when they needed to access codepoints
outside the BMP.  This was a hack, but the Java, JavaScript, and .Net
folks went with it in order to make their implementations a little easier.
 Some newer languages are using UTF8 as their internal String
representation, which works ok.  Note that counting code points correctly
in either system requires a full scan of the string.  UTF32 doesn't suffer
from this, but always requires 4 bytes per code point.

-- 
Joe Hildebrand

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 07 Jun 2013 19:24:39 +0900

On 2013/06/06 23:57, Tim Bray wrote:
> F0, 90, 8D, 86
> On Thu, Jun 6, 2013 at 4:15 AM, Douglas Crockford<[email-masked]>wrote:
>

I fully agree with what Tim says above: We know (and to a certain extent 
have to accept) that there are implementations that, surely way more by 
accident than by any kind of intent, send unpaired surrogates. But we 
should try to do whatever we can in the spec to make it perfectly clear 
that there are no good reasons whatsoever to actually do that.

Although we may not end up with exactly parallel or equivalent language, 
I think the situation is fairly similar to the one regarding duplicate 
keys: The current spec isn't totally clear, and in practice it happens, 
and for some implementations, it may be an unreasonable burden to 
require to check everything on sending, but it's not something that one 
would or should do on purpose.

A second point:

Just to get the correct definitions from the Unicode side, here are the 
easiest references, everything on a page:

http://www.unicode.org/glossary/

In a few of my mails today, I have also used "code point", but I didn't 
intend to include surrogates.

There's a better term, namely Unicode Scalar Value 
(http://www.unicode.org/glossary/#unicode_scalar_value):

Any Unicode code point except high-surrogate and low-surrogate code 
points. In other words, the ranges of integers 0 to D7FF16 and E00016 to 
10FFFF16 inclusive. (See definition D76 in Section 3.9, Unicode Encoding 
Forms.)

Regards,    Martin.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 07 Jun 2013 12:42:17 +0200

>that there are no good reasons whatsoever to actually do that.

Actually there are many good reasons for having unpaired surrogates in
JSON documents. A simple example would be a test suite for string APIs.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 Jun 2013 08:56:42 -0700

<no hat>

This may be a part of the spec where some people have to hold their noses. The Unicode definition of "character" does not include non-characters, and the code points for some of those non-characters make sense in JSON strings when those strings. Bjoern has pointed out a good one: strings used for test cases of other code. The issue not just unpaired surrogates. Do we *really* want to prohibit:
   { "End of data marker": "\uFFFF" }

Proposal:

Remove the word "character" from the spec except in an explanatory paragraph in Section 2.5 that says:
   All code points, even those that represent non-characters in the Unicode specification [UNICODE], are allowed in JSON strings.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 09:01:39 -0700

On Fri, Jun 7, 2013 at 8:56 AM, Paul Hoffman <[email-masked]> wrote:

> This may be a part of the spec where some people have to hold their noses.
> The Unicode definition of "character" does not include non-characters, and
> the code points for some of those non-characters make sense in JSON strings
> when those strings. Bjoern has pointed out a good one: strings used for
> test cases of other code. The issue not just unpaired surrogates. Do we
> *really* want to prohibit:

Yes, I *really* want to prohibit that. The one corner case it buys you is
outweighed by a factor of a thousand or so in not being able to use
general-purpose string processing software to deal with JSON payloads.
BTW, a huge amount of deployed software out there ALREADY processes JSON
text fields using general-purpose string processing libraries, and will
explode unpredictably and in hard-to-debug ways if this starts happening.

Also, consider the lovely consequences when unpaired surrogates start
showing  up in key fields and are fed to hash functions in every
programming language in the world, which expect to receive Unicode
characters.

 -T

>    All code points, even those that represent non-characters in the
> Unicode specification [UNICODE], are allowed in JSON strings.
>
> --Paul Hoffman

---

From: Stephen Dolan <[email-masked]>
Date: Fri, 7 Jun 2013 17:09:46 +0100

I think it is useful to distinguish three cases of codepoint:
 (1) Those which are valid characters in a particular Unicode revision
 (2) Those which are unallocated codepoints which may become valid
characters in a later Unicode revision
 (3) The noncharacter codepoints which will never be valid

(3) includes such beasts as U+FFFE (which you can only get by reading
a UTF16 byte order mark with the wrong byte order). The set (1)
increases with every Unicode revision to include characters from (2),
but (3) is stable (see
http://unicode.org/policies/stability_policy.html).

I think JSON should allow characters from (1) and (2) to avoid being
dependent on a specific Unicode revision. I do not think (3) should be
allowed - this would cause problems with many existing parsers which
represent JSON strings using another system's native unicode
representation.

The argument about testsuites does not seem compelling, as any such
testsuite testing behaviour of string functions with bad Unicode would
also include invalidly-encoded Unicode (such as overlong UTF8
sequences) which cannot be represented at all in JSON, even with
escaping.

Stephen

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 18:15:45 +0200

On 2013-06-07 18:01, Tim Bray wrote:
> On Fri, Jun 7, 2013 at 8:56 AM, Paul Hoffman ... wrote:
>
>     This may be a part of the spec where some people have to hold their
>     noses. The Unicode definition of "character" does not include
>     non-characters, and the code points for some of those non-characters
>     make sense in JSON strings when those strings. Bjoern has pointed
>     out a good one: strings used for test cases of other code. The issue
>     not just unpaired surrogates. Do we *really* want to prohibit:
> Yes, I *really* want to prohibit that. The one corner case it buys you
> is outweighed by a factor of a thousand or so in not being able to use

and what about { "Decorate my slash": "\/" } and "general-purpose string 
processing software". Isn't this also a case, where you need a 
"pre-conditioner" that replaces the JSON specific escape sequence "\" 
with "/" before feeding it into "general-purpose string processing 
software" :-?)

> programming language in the world, which expect to receive Unicode

For today I better not imagine all these laguages and implementations 
blindly stuffing some json text transformed into their own memory 
structures ... maybe later during the weekend

>     ...

Stefan.

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 09:18:07 -0700

I agree 100% with Stephen Dolan.  -T

On Fri, Jun 7, 2013 at 9:09 AM, Stephen Dolan <[email-masked]>wrote:

> non-characters, and the code points for some of those non-characters make
> sense in JSON strings when those strings. Bjoern has pointed out a good
> one: strings used for test cases of other code. The issue not just unpaired
> surrogates. Do we *really* want to prohibit:

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 18:18:42 +0200

Sorry, did miss one slash in my text, whci should irritate the reader.
c.f. below.
> "pre-conditioner" that replaces the JSON specific escape sequence "\"

above lines should of course read:

"pre-conditioner" that replaces the JSON specific escape sequence "\/"

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 09:19:41 -0700

On Fri, Jun 7, 2013 at 9:15 AM, Stefan Drees <[email-masked]> wrote:

> "pre-conditioner" that replaces the JSON specific escape sequence "\" with
> "/" before feeding it into "general-purpose string processing software" :-?)

Red herring.  JSON, just like XML, has ways to encode characters that are
hard to type.  What we're arguing about is the actual content of the
payload after the parser/pre-conditioner.  -T

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 13:08:57 -0400

"Martin J. Dürst" scripsit:

> But we should try to do whatever we can in the spec to make it perfectly
> clear that there are no good reasons whatsoever to actually do that.

That's a little stronger than the truth.  You can use Java(Script)
strings, which are sequences of arbitrary 16-bit code units, to
represent an immutable vector of unsigned integers bounded by 65535,
and I have actually done so.  However, it is not a good idea to do so
in open interchange.

> points. In other words, the ranges of integers 0 to D7FF16 and E00016
> to 10FFFF16 inclusive. (See definition D76 in Section 3.9, Unicode
> Encoding Forms.)

Yes, we should use either "Unicode scalar value" or "Unicode 16-bit code
unit", depending on what we decide on.

-- 
That you can cover for the plentiful            John Cowan
and often gaping errors, misconstruals,         http://www.ccil.org/~cowan
and disinformation in your posts                [email-masked]
through sheer volume -- that is another misconception.  --Mike to Peter

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 13:19:51 -0400

> general-purpose string processing software to deal with JSON payloads.

Most general-purpose string processing software is perfectly happy
with U+FFFF.  There are three different kinds of code points here,
and it doesn't help to conflate them:

1) Surrogate code points.  These will never be assigned to any characters,
and reserved for use as UTF-16 code units.  There are exactly 2048 of
these, from U+DC00 to U+DFFF.

2) Non-character code points.  These will never be assigned to any
characters, and are not meant to be interchanged, but internal software
is expected to handle them.  There are exactly 66 of these, and U+FFFF
is one.  See <http://www.unicode.org/faq/private_use.html#noncharacters>
for more about this group.

3) Unassigned code points.  These are not assigned to any characters
today, but may be assigned in future.  They may be interchanged.
Internal libraries should process them.

My view is that group 1 are and should be disallowed in JSON; others
disagree.  Group 2 should be avoided by JSON creators, but accepted by
JSON parsers, which may choose to change them to U+FFFD (replacement
character).  Group 3 are and should be valid in JSON.

-- 
Values of beeta will give rise to dom!          John Cowan
(5th/6th edition 'mv' said this if you tried    http://www.ccil.org/~cowan
to rename '.' or '..' entries; see              [email-masked]
http://cm.bell-labs.com/cm/cs/who/dmr/odd.html)

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 13:20:22 -0400

Stefan Drees scripsit:

> and what about { "Decorate my slash": "\/" } and "general-purpose
> string processing software". Isn't this also a case, where you need
> a "pre-conditioner" that replaces the JSON specific escape sequence
> "\" with "/" before feeding it into "general-purpose string
> processing software" :-?)

That would be the JSON parser.

-- 
We pledge allegiance to the penguin             John Cowan
and to the intellectual property regime         [email-masked]
for which he stands, one world under            http://www.ccil.org/~cowan
Linux, with free music and open source
software for all.               --Julian Dibbell on Brazil, edited

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 10:25:59 -0700

John is right and I am wrong.  What I really want to legislate against is:

{
 "Mis-use of UTF-16 surrogates" : "\udf46\ud800AB\udf11CD\ud812EF",
 "Mis-use of flipped BOM" : "AB\ufffeCD"
}

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 13:30:56 -0400

>  "Mis-use of flipped BOM" : "AB\ufffeCD"

FFEF/FFFE only have their magic meanings at the beginning of a text;
otherwise, FFFE and FFFF are formally equivalent (they are both
non-characters, my group 2).

-- 
Do what you will,                       John Cowan
   this Life's a Fiction                [email-masked]
And is made up of                       http://www.ccil.org/~cowan
   Contradiction.  --William Blake

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 12:52:04 -0500

> practice, which is why it's pretty safe to do the update).

We already know to allow the user of unassigned code points.  We can't
avoid having a normative reference to a Unicode version, if that was
your goal.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 13:09:47 -0500

> JSON documents. A simple example would be a test suite for string APIs.

Or what heck, if ECMAScript allows any 16-bit values, 0x0000..0xFFFF
to be used (escaped as \uXXXX if necessary) then one very useful use
of that is encoding binary data: when parsing you know if you have
binary data when you see any 16-bit code units that don't make any
sense in Unicode text.  Not that I'm advocating this... but if we did
allow this then it wouldn't preclude us from saying that a string of
text must not include unpaired surrogates.

Nico
--

---

From: Tatu Saloranta <[email-masked]>
Date: Fri, 7 Jun 2013 11:15:47 -0700

> text must not include unpaired surrogates.

Not really. Since most content is exchanged as UTF-8, it gives you 3-to-2
conversion, which is no better than Base64 (4-to-3), nor necessarily any
faster. It would only help when using UTF-16.
APIs would expose this as text, requiring conversion or coercion (depending
on platform), so it would not even be a short-cut.

-+ Tatu +-

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 18:27:42 +0000

>avoid having a normative reference to a Unicode version, if that was

"character" implies "assigned", I think.  We need a Unicode ref, and we
need to ensure that the text allows the use of code points unassigned in
that version we refer to, which isn't clear in 4627.

-- 
Joe Hildebrand

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 07 Jun 2013 20:40:51 +0200

>represent JSON strings using another system's native unicode

Please see <http://www.unicode.org/versions/corrigendum9.html>.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Fri, 7 Jun 2013 11:49:37 -0700

Interestingly, that doesn’t mention surrogates. -T

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 15:48:04 -0400

> avoid having a normative reference to a Unicode version, if that was

As long as it's recent enough to incorporate non-characters, that's enough.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is
dramatically overdescribed.  Still other languages are simultaneously
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King

---

From: Nico Williams <[email-masked]>
Date: Fri, 7 Jun 2013 18:22:59 -0500

> Interestingly, that doesn’t mention surrogates. -T

I think the only reasonable interpretation of surrogates is as pairs.
An incomplete pair would be an error, and surrogates in any UTF other
than UTF-16 would also be an error -- in those cases we're not even
talking about noncharacter code points.

But corrigendum #9 does make me think that the best thing to do is to
pass them through, as with any other noncharacters and private-use
code points.  If nothing else it's one less thing to check for in a
parser.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 23:06:20 -0400

> code points.  If nothing else it's one less thing to check for in a

If the parser provides an internal representation other than UTF-16,
however, that just isn't possible.

-- 
John Cowan <[email-masked]>             http://www.ccil.org/~cowan
Awk!" sed Grep. "A fscking python is perloining my Ruby; let me bash
    him with a Cshell!  Vi didn't I mount it on a troff?" --Francis Turner

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 08 Jun 2013 10:47:42 +0200

>however, that just isn't possible.

Unpaired surrogate code points cannot be represented in UTF-16 and you
can easily use e.g. `UTF-32-but-allow-unpaired-surrogates` internally.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 8 Jun 2013 11:38:25 +0200

>   All code points, even those that represent non-characters in the Unicode specification [UNICODE], are allowed in JSON strings.

A much better way to handle this problem would be

"For the purposes of this specification, the term "character" includes both Unicode characters and ______."

Fill in the blanks based on what we want to do here:

(1) I think it is now clear that we don't want to shun Unicode non-characters (see corrigendum #9).

(2) I think it is also clear that we want to include all Unicode code positions that could become Unicode characters in future versions of Unicode according to the compatibility spec.

(3) I also think that we don't want to change UTF-8, the UTF-16s, or the UTF-32s, so what's possible in "native" encoding will be controlled by those specifications, minus ", \, the C0 characters.

(4) The remaining question appears what we do with unpaired escaped surrogates.
The answer will have to be a bit wishy-washy, because anything strong will invalidate half of the implementations.  If is probably a good idea not to "break" those applications that compensate JavaScript's lack of a binary string type by using UTF-16 as a vector of unconstrained 16-bit values, but we also cannot mandate that everyone adopt this 1990s style hack.

Is my conclusion correct that, at the model level, it's either 0x0..0x10FFFF (including unpaired surrogates) or 0x0..0xD7FF+0xE000..0x10FFFF (without unpaired surrogates, i.e. the Unicode scalar values)?  Parts of this range will only be accessible by escape sequences (", \, C0, and, when allowed, unpaired surrogates).

Grüße, Carsten

---

From: Stefan Drees <[email-masked]>
Date: Sat, 08 Jun 2013 11:56:41 +0200

On 2013-06-07 18:19, Tim Bray wrote:
> On Fri, Jun 7, 2013 at 9:15 AM, Stefan Drees ... wrote:
>
> Red herring.  JSON, just like XML, has ways to encode characters that
> are hard to type.  What we're arguing about is the actual content of the
> payload after the parser/pre-conditioner.  -T ...

I beg to differ. I probed the claim, that "general-purpose string 
processing software" would have problems with entity so-and-so but not 
with JSON text as is.

Also, when it comes to "typing" (as human-machine directed communication 
means) forward slashes, dot-ini files or YAML text in my experience are 
far more easy to type than JSON text ;-)

As John Cowan rightfully stated: "That[the general-purpose string 
processing software] would be the JSON parser." **but** it's task may be 
considered being anything from easy to hard.
We discuss here on these matters, so it's not a red herring, isn't it?

Personal motivation (if time and interest allow):

Any differences in expressing common "atomic" values or "core 
structures" (like type of inter atomic bindings) should be verified as 
each may introduce otherwise not needed hurdles or even traps in 
translation. It is data interchange we specify.

For a transformation source and target that look quite similar (as eg. 
JSON variables and corresponding language L "things") for some language 
L in (ECMAScript, PHP, Pyton, ...) one would expect, to not need any 
"heavy" lifting when transforming from one representation into another.

I observe, that "we programmers" more and more look at our code (esp. 
atomic variables and structures in some language L), compare these to 
the serialized (prettyprinted) JSON equivalent and then expect them to 
be essentially identical (some people even start to think eg. "JSON 
**is** Python syntax" as recently observed on the python-ideas mailing 
list when argueing in favour of a JSON serializer).

At least one should expect for a successful and proven data interchange 
format, that the atomic values simply "slip back and forth" and do not 
need special per-character handling.

So the more we extend basic escaping "C"ulture (as we further superset 
the C escape sequences), the wider this gap will become.

Interesting "structural" rule differences (not being candidate to 
change, I know) are:

* the trailing comma allowed (or even needed) in many programming
   languages for instances of list-like structures being not allowed in
   JSON objects nor arrays (I am sorry that this will not change)

* the JSON need for quoting the names (I am neutral here)

do bite us "machine instructing humans" once in a while, don't they?

The "might-look-like-a-babylonian" system of programming languages that 
evolved as solution attempts to the

* N humans instructing
* M machines with
* L languages utilizing
* F formats for data interchange or storage to handle
* P problems

is hopefully offering NP-complete solutions :-) but IMO some of the 
couplings should be reconsidered with passionate caution from time to time.

Maybe now is such a "time" to reconsider some regions of the space 
spanned by the cartesian product L x (F=JSON).

Thanks for reading this long opinionated message (which I also did 
before sending. Promised ;-)

Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 8 Jun 2013 08:25:27 -0700

> The answer will have to be a bit wishy-washy, because anything strong will invalidate half of the implementations.  If is probably a good idea not to "break" those applications that compensate Java…

Ummm, how is that "much better"? "Code points minus THEONESWEHATE" seems a lot simpler than "characters plus ADDITIONAL1 plus ADDITIONAL2 plus ADDITIONAL3".

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 8 Jun 2013 18:07:11 +0200

> Ummm, how is that "much better"? "Code points minus THEONESWEHATE" seems a lot simpler than "characters plus ADDITIONAL1 plus ADDITIONAL2 plus ADDITIONAL3".

Well, we need a (short) word for "that thing that goes into strings".

Most people I know would expect a "string" to be composed of "characters".

So why not define the "character" in such a way that it works for us?

RFC 4627 does not go to the length to formally define its terms.  In particular, it is not always clear where a term is imported from somewhere else, and what that somewhere is.  This provides some potential for improvement.  Once this is done, there is no need to use the term character exactly like Unicode uses it.  (The term "Unicode character" can be used where that specific definition is actually meant.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Sat, 8 Jun 2013 15:41:43 -0500

>> The answer will have to be a bit wishy-washy, because anything strong will invalidate half of the implementations.  If is probably a good idea not to "break" those applications that compensate Jav…

I'm starting to conclude that "anything goes..."

"...with the understanding that in some cases some, er, code points
get dropped".  E.g., a surrogate encoded in UTF-8 or UTF-32 makes no
sense, and though some parsers can leave it be, a parser that converts
to UTF-16 probably can't leave there.  Note that there's a bit of a
difference between an escaped and an unescaped such code point: the
former might be an attempt to pass binary data, while the latter could
just be either an error or the result of unescaping the first.

> Ummm, how is that "much better"? "Code points minus THEONESWEHATE" seems a lot simpler than "characters plus ADDITIONAL1 plus ADDITIONAL2 plus ADDITIONAL3".

The latter seems better in that we defer the definition of "character"
to the UC, but the former seems better in that it's much more specific
about THEONESWEHATE.  The latter seems likely to allow more
interoperability with any implementations that allow just about
anything.

Nico
--