Subject: [Json] Proposal for strings/Unicode text

From: Tim Bray <[email-masked]>
Date: Wed, 12 Jun 2013 11:46:54 -0700

Rationale:
- emphasize the important fact that Strings are *intended* for Unicode
characters
- document the important fact that the rules allow horrible Unicode
practices
- say “backslash” instead of “reverse solidus” :)

In section 1, introduction

Before:
   A string is a sequence of zero or more Unicode characters [UNICODE].
After:
   A string is intended to contain sequences of zero or more Unicode
characters [UNICODE 6.2]

Rewrite section 2.5 as follows:

Strings begin and end with quotation marks.  They are intended,to contain
sequences of Unicode characters; Note however that the ABNF in this section
allows the inclusion of 16-bit quantities in ways which can never be useful
for representing characters and is likely to cause breakage in software
designed to process Unicode text.

The ABNF allows the use of many Unicode code points that could be used in
future to represent Unicode characters, but have not yet been assigned.
Therefore, this specification should not need revision as the Unicode
character repertoire continues to grow.

16-bit quantities (normally Unicode characters from the Basic Multingual
Pane(U+0000 through U+FFFF) may be “escaped”, or represented as a
six-character sequence: a backslash (U+005C REVERSE SOLIDUS), followed  by
the lowercase letter u, followed by four hexadecimal digits that encode the
character's code point.  The hexadecimal letters A though F can be upper or
lower case.  So, for example, a string containing only a single backslash
may be represented as "\u005C".

Alternatively, there are two-character sequence escape representations of
some popular characters.  So, for example, a string containing only a
single backslash may be represented more compactly as "\\".

 To escape an extended character that is not in the Basic Multilingual
Plane, the character is represented as a twelve-character sequence,
encoding the UTF-16 surrogate pair.  So, for example, a string containing
only U+1D11E G CLEF may be represented as
   "\uD834\uDD1E".

=== insert ABNF here ====

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 13 Jun 2013 00:30:10 +0200

Hmm.

Somehow I think the JSON specification should focus on describing what the intended usage is.

I strongly prefer adding an appendix M for things that you can do with the ABNF that are almost, but not entirely unlike JSON.

Grüße, Carsten

PS.: I support jettisoning liturgical language in standards, and I applaud Douglas for slipping with respect to the liturgical term "octet" only twice (both in the same paragraph).  But a document must also speak the language of its editor, and if Douglas thinks "reverse solidus" is the best way to speak about what some Germans (but never me) would call "Rückschräger", that's fine, as long as it is consistent.

PPS.: On the specific wording:
>    A string is intended to contain sequences of zero or more Unicode characters [UNICODE 6.2]

A string is a sequence of characters.  [not sequences of them]
Add something like: "To reduce the burden on implementations, JSON is less selective in what it accepts as a character than Unicode itself is.  See also Appendix M."

> Strings begin and end with quotation marks. 

The representation does, the string rarely does.
RFC4627 got this right consistently, but in a tight language where extracting a single sentence may lose the necessary context.

> They are intended,to contain sequences of Unicode characters; Note however that the ABNF in this section allows the inclusion of 16-bit quantities in ways which can never be useful for representing characters and is likely to cause breakage in software designed to process Unicode text.

This is where I would simply point to Appendix M.

> The ABNF allows the use of many Unicode code points that could be used in future to represent Unicode characters, but have not yet been assigned. Therefore, this specification should not need revision as the Unicode character repertoire continues to grow.

This is something that even could be said in the introduction.
Or in a section about stability and protocol evolution (the same section that is needed to say that [past and future] changes in JavaScript don't change JSON).

> 16-bit quantities

These are Unicode code points.

> (normally Unicode characters from the Basic Multingual Pane(U+0000 through U+FFFF) may be “escaped”, or represented as a six-character sequence: a backslash (U+005C REVERSE SOLIDUS), followed  by the lowercase letter u, followed by four hexadecimal digits that encode the character's code point.  The hexadecimal letters A though F can be upper or lower case.  So, for example, a string containing only a single backslash may be represented as "\u005C".
> 
> Alternatively, there are two-character sequence escape representations of some popular characters.  So, for example, a string containing only a single backslash may be represented more compactly as "\\".
> 
>  To escape an extended character

Non-BMP characters aren't "extended".  They have rights, too!
You MUST rid your mind of discriminating against them.
(I know, this was just copied over...)

> that is not in the Basic Multilingual Plane, the character is represented as a twelve-character sequence, encoding the UTF-16 surrogate pair.  So, for example, a string containing only U+1D11E G CLEF may be represented as
>    "\uD834\uDD1E".

(This could add a small apologetic clause pointing out the UTF-16 roots of the weird notation.  Or not.)
This needs another pointer to appendix M.

---

From: Norbert Lindenberg <[email-masked]>
Date: Wed, 12 Jun 2013 15:46:07 -0700

On Jun 12, 2013, at 11:46 , Tim Bray wrote:

> Rationale:
> - emphasize the important fact that Strings are *intended* for Unicode characters
> - document the important fact that the rules allow horrible Unicode practices

The JSON RFC seems to use Unicode character names, in this case case "reverse solidus".

> Strings begin and end with quotation marks.  They are intended,to contain sequences of Unicode characters; Note however that the ABNF in this section allows the inclusion of 16-bit quantities in ways which can never be useful for representing characters and is likely to cause breakage in software designed to process Unicode text.

This warning is too vague to be useful. Which specific risks do you think need to be discussed here? Also, the ABNF doesn't do anything specifically for 16-bit quantities, as far as I can see.

> 16-bit quantities (normally Unicode characters from the Basic Multingual Pane(U+0000 through U+FFFF) may be “escaped”, or represented as a six-character sequence: a backslash (U+005C REVERSE SOLIDUS), followed  by the lowercase letter u, followed by four hexadecimal digits that encode the character's code point.  The hexadecimal letters A though F can be upper or lower case.  So, for example, a string containing only a single backslash may be represented as "\u005C".

These escape sequences aren't about 16-bit quantities - they represent Unicode BMP code points. If the parser inserts a BMP code point into an output string in UTF-16 (in JavaScript, Java, and others), then the result will be a 16-bit quantity. If it inserts the code point into an output string in UTF-8 (in Python and others), then the result will be a sequence of 1-3 bytes.

If we discuss these escape sequences in prose, then the sequences for BMP and supplementary characters need to be discussed together so that it's clear what's a surrogate pair and how unpaired surrogates are handled.

JSON syntax allows almost all Unicode code points (with the exceptions visible in the "unescaped" production) to be inserted into strings directly, so escape sequences are mostly a convenience for using JSON in environments that restrict the set of allowed characters (such as RFCs), don't have the necessary fonts and input methods installed, or benefit from making the code points visible (e.g., in test cases).

> Alternatively, there are two-character sequence escape representations of some popular characters.  So, for example, a string containing only a single backslash may be represented more compactly as…

I'd assume that this is not about popularity, but about the need to represent control characters and characters that are also used within the JSON syntax. I don't see two-character escapes for "e" or "的".

Norbert

---

From: Tim Bray <[email-masked]>
Date: Wed, 12 Jun 2013 15:49:56 -0700

Revised per Carsten's  and Norbert’s input. I’m unconvinced about Appendix
M since I think the “16-bit quantities” para says what needs saying, but he
caught a few bugs.

Before:
   A string is a sequence of zero or more Unicode characters [UNICODE].
After:
   A string is intended to contain a sequence of zero or more Unicode
characters [UNICODE 6.2]

Strings are delimited  with quotation marks (U+0022 QUOTATION MARK).  They
are intended,to contain sequences of Unicode characters.  Note however that
the normative ABNF in this section allows the inclusion of 16-bit
quantities, for example unpaired surrogate-block code points, in ways which
can never be useful for representing characters and is likely to cause
breakage in software designed to process Unicode text.

16-bit quantities, normally representing Unicode characters from the Basic
Multingual Pane (U+0000 through U+FFFF), may be “escaped”, or represented
as a six-character sequence: a backslash (U+005C REVERSE SOLIDUS),
followed  by the lowercase letter u, followed by four hexadecimal digits
that encode the character's code point.  The hexadecimal letters A though F
can be upper or lower case.  So, for example, a string containing only a
single backslash may be represented as "\u005C".

 To escape a non-BMP character that is not in the Basic Multilingual Plane,
the character is represented as a twelve-character sequence, encoding the
UTF-16 surrogate pair.  So, for example, a string containing only U+1D11E G
CLEF may be represented as
   "\uD834\uDD1E".

On Wed, Jun 12, 2013 at 3:30 PM, Carsten Bormann <[email-masked]> wrote:

> I strongly prefer adding an appendix M for things that you can do with the
> ABNF that are almost, but not entirely unlike JSON.
>
> Grüße, Carsten
>
> PS.: I support jettisoning liturgical language in standards, and I applaud
> Douglas for slipping with respect to the liturgical term "octet" only twice
> (both in the same paragraph).  But a document must also speak the language
> of its editor, and if Douglas thinks "reverse solidus" is the best way to
> speak about what some Germans (but never me) would call "Rückschräger",
> that's fine, as long as it is consistent.
>
> Add something like: "To reduce the burden on implementations, JSON is less
> selective in what it accepts as a character than Unicode itself is.  See
> also Appendix M."
>
> RFC4627 got this right consistently, but in a tight language where
> extracting a single sentence may lose the necessary context.
>
> > They are intended,to contain sequences of Unicode characters; Note
> however that the ABNF in this section allows the inclusion of 16-bit
> quantities in ways which can never be useful for representing characters
> and is likely to cause breakage in software designed to process Unicode
> text.
>
> > The ABNF allows the use of many Unicode code points that could be used
> in future to represent Unicode characters, but have not yet been assigned.
> Or in a section about stability and protocol evolution (the same section
> that is needed to say that [past and future] changes in JavaScript don't
> change JSON).
>
> > 16-bit quantities
>
> > (normally Unicode characters from the Basic Multingual Pane(U+0000
> through U+FFFF) may be “escaped”, or represented as a six-character
> sequence: a backslash (U+005C REVERSE SOLIDUS), followed  by the lowercase
> letter u, followed by four hexadecimal digits that encode the character's
> code point.  The hexadecimal letters A though F can be upper or lower case.
>  So, for example, a string containing only a single backslash may be
> represented as "\u005C".
> >
> > Alternatively, there are two-character sequence escape representations
> of some popular characters.  So, for example, a string containing only a
> > that is not in the Basic Multilingual Plane, the character is
> represented as a twelve-character sequence, encoding the UTF-16 surrogate
> pair.  So, for example, a string containing only U+1D11E G CLEF may be
> represented as
> >    "\uD834\uDD1E".
>

---

From: Tim Bray <[email-masked]>
Date: Wed, 12 Jun 2013 15:54:25 -0700

> The JSON RFC seems to use Unicode character names, in this case case

We’re not allowed to change JSON but we are allowed to improve the spec.
The term “solidus” is obscure and off-putting.

This warning is too vague to be useful. Which specific risks do you think
> need to be discussed here? Also, the ABNF doesn't do anything specifically
> for 16-bit quantities, as far as I can see.
>

Agreed; I introduced a specific example.

>  > 16-bit quantities (normally Unicode characters from the Basic
> Multingual Pane(U+0000 through U+FFFF) may be “escaped”, or represented as
> a six-character sequence: a backslash (U+005C REVERSE SOLIDUS), followed
>  by the lowercase letter u, followed by four hexadecimal digits that encode
> the character's code point.  The hexadecimal letters A though F can be
> upper or lower case.  So, for example, a string containing only a single
> backslash may be represented as "\u005C".
>
> These escape sequences aren't about 16-bit quantities - they represent
> Unicode BMP code points.

They also represent surrogates.  Thus they combine fish and bicycles, and I
think the only accurate way to describe what you can escape is “16-bit
quantities”

> If we discuss these escape sequences in prose, then the sequences for BMP
> and supplementary characters need to be discussed together so that it's
> clear what's a surrogate pair and how unpaired surrogates are handled.
>

Hm?  I see no point in replicating the discussion of surrogates in Unicode,
and there is nothing useful to say about the handling of unpaired
surrogates.

> I'd assume that this is not about popularity, but about the need to
> represent control characters and characters that are also used within the
> JSON syntax. I don't see two-character escapes for "e" or "的".
>

I tried to stay close to the language of the original.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 13 Jun 2013 01:40:49 +0200

On Jun 13, 2013, at 00:49, Tim Bray <[email-masked]> wrote:

> Strings are delimited  with quotation marks (U+0022 QUOTATION MARK).  They are intended,to contain sequences of Unicode characters.  Note however that the normative ABNF in this section allows the inclusion of 16-bit quantities, for example unpaired surrogate-block code points, in ways which can never be useful for representing characters and is likely to cause breakage in software designed to process Unicode text.

A kitchen knife is intended to cut food.  Note however that the normative cutting edge of the knife also allows the use against intruding burglars, for example by slashing open their guts, which can never be useful for food and is likely to cause you trouble in a number of ways, including bloodstains on your shirt.

I don't want to read this sentence in the introduction of the manual for my kitchen knife, please.

Again, can we focus on the correct usage and ban avenues for misuse to an appendix?
It could even be called "the JSON character model" to obscure its grisly purpose.

You are not going to get both "correct" and "understandable" if you combine the intended usage with all the arcane legacy issues into one piece of text.

(If I'm the only one who cares about this editorial issue, this is going to be my last comment on this... if this WG does manage to turn one of the nicest-written RFCs into goulash, I can always point my students to read the original.)

Grüße, Carsten

PS.: "16-bit quantities" is highly confusing.  Outside of mathematics and physics, "quantity" gives the amount of something.  There is absolutely no reason not to use the term "code point" from Unicode.

»In the Unicode Standard, the codespace consists of the integers from 0 to 10FFFF[16], comprising 1,114,112 code points available for assigning the repertoire of abstract characters.«

See also Table 2-3.

Yes, all of them can be used in a JSON document...  
Some of them must be escaped, some of them are non-characters but otherwise innocuous, some of them (surrogates) actively try to slash your guts, but they are all code points.

»Surrogate code points cannot be conformantly interchanged using Unicode encoding forms. They do not correspond to Unicode scalar values and thus do not have well-formed representations in any Unicode encoding form. (See Section 3.8, Surrogates.)«

So they must be escaped (unless you are using them for their intended purpose in a UTF-16-encoded JSON document, a rare species).  There also is no way to notate a sequence of a high surrogate and a low surrogate using escape sequences in JSON, because that always stands for the non-BMP code point that results from the values of the two surrogates.  All that should be mentioned somewhere, because it takes a couple of hours of analysis to ascertain that.  (Maybe also mention that that doesn't hurt the "JavaScript string as vector of 16-bit numbers" use case, because the non-BMP characters can be (will be!) taken apart again by the recipient.  So you don't even have to escape them...  More stuff for the appendix, or maybe for a special "worst practice" document.)

---

From: R S <[email-masked]>
Date: Wed, 12 Jun 2013 16:56:28 -0700

> Again, can we focus on the correct usage and ban avenues for misuse to an

The document should refrain from value judgements. The "correct" use is
what JSON is already being used for. Even Tim's "breakage" language is a
little strong (although I can live with it).

- Rob

---

From: John Cowan <[email-masked]>
Date: Wed, 12 Jun 2013 20:32:13 -0400

Norbert Lindenberg scripsit:

> JSON syntax allows almost all Unicode code points (with the exceptions
> visible in the "unescaped" production) to be inserted into strings
> directly, so escape sequences are mostly a convenience for using JSON
> in environments that restrict the set of allowed characters (such as
> RFCs), don't have the necessary fonts and input methods installed,
> or benefit from making the code points visible (e.g., in test cases).

Officially, yes.  But surrogate code points cannot be inserted directly
if the representation is UTF-8 (otherwise it becomes CESU-8 instead)
or UTF-16 (otherwise it is broken UTF-16) or random non-Unicode encodings.
So UTF-32 is the only encoding into which a surrogate code point can be
inserted directly -- and nobody uses it.

In practice, therefore, code points in the range D800-DFFF must be escaped
if they are to be used.

-- 
Mark Twain on Cecil Rhodes:                    John Cowan
I admire him, I freely admit it,               http://www.ccil.org/~cowan
and when his time comes I shall                [email-masked]
buy a piece of the rope for a keepsake.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 13 Jun 2013 12:15:18 +0200

>inserted directly -- and nobody uses it.

http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf

  D31 UTF-32 encoding form:

  The Unicode encoding form which assigns each Unicode scalar value to a
  single unsigned 32-bit code unit with the same numeric value as the
  Unicode scalar value.

    • In UTF-32, the code point sequence <004D, 0430, 4E8C, 10302> is
      represented as <0000004D 00000430 00004E8C 00010302>.

    • Because surrogate code points are not included in the set of 
      Unicode scalar values, UTF-32 code units in the range
      0000D800_16 .. 0000DFFF_16 are ill-formed.

    • Any UTF-32 code unit greater than 0010FFFF_16 is ill-formed.

Note the second bullet point.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: John Cowan <[email-masked]>
Date: Thu, 13 Jun 2013 08:16:20 -0400

>       0000D800_16 .. 0000DFFF_16 are ill-formed.

Well, sure.  Note the fine distinction between "can" (in physical
fact) and "may" (in the RFC 2119 sense).  It's invalid to have unpaired
surrogates in *any* context.  But at least it is safe and possible to do
so in UTF-32.  In UTF-8, there is no representation at all, and in UTF-16
you can't tell the difference between two consecutive unpaired surrogates
of opposite polarities and a surrogate pair.  (Though come to think of
it, escaping doesn't allow two consecutive unpaired surrogates either,
so maybe we can fairly say that either UTF-16 or UTF-32 allow them.)

The point is that if JSON is encoded in UTF-8, any surrogate code points
MUST be escaped, even though the grammar does not say so.

-- 
John Cowan            http://www.ccil.org/~cowan     [email-masked]
Uneasy lies the head that wears the Editor's hat! --Eddie Foirbeis Climo

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 13 Jun 2013 17:46:32 +0000

>MUST be escaped, even though the grammar does not say so.

What about changing the grammar to make that clear?

unescaped = %x20-21 / %x23-5B / %x5D-%xD7FF / %E000-%10FFFF
 ; Any unicode code point except control characters, QUOTATION MARK,
 ; REVERSE SOLIDUS, or code points reserved for UTF-16 surrogates

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Thu, 13 Jun 2013 11:00:19 -0700

> MUST be escaped, even though the grammar does not say so.

Why?  UTF-8 is perfectly capable of representing those integers.  Yes, the
spec says that You Shouldn’t Do That, but it says the same thing about
unpaired surrogates in UTF-16.  For historical reasons JSON allows the
encoding of stuff that is strictly nonconforming to Unicode.  This will
break lots of things, not just UTF-8 decoders (most of which, I bet, will
never actually notice).  -T

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Nico Williams <[email-masked]>
Date: Thu, 13 Jun 2013 13:16:08 -0500

> never actually notice).  -T

I was thinking the same thing.  We're way into "you shouldn't do that
but actually, look, you can" territory.

I think it's important that some \uXXXX sequences not be unescaped by
intermediaries, or, rather, that certain code points always be escaped
by encoders.  E.g., \u0000, \u2028, \uD800, and so on.  The reason is
that a parser may not be able to handle such code points unescaped.
We should stop at that and otherwise allow just about any code point.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Thu, 13 Jun 2013 14:19:55 -0400

Tim Bray scripsit:

> unpaired surrogates in UTF-16.  

If this is going to be just another informational RFC, then of course
we can do what we want.  But if it's a standards-track RFC, it has to
play nicely with other standards-track RFCs.  And RFC 3629 aka STD 63 is
not just standards-track, it's an Internet Standard, it's ten years old,
and it says uncompromisingly:

   The definition of UTF-8 prohibits encoding character numbers between
   U+D800 and U+DFFF, which are reserved for use with the UTF-16
   encoding form (as surrogate pairs) and do not directly represent
   characters.  When encoding in UTF-8 from UTF-16 data, it is necessary
   to first decode the UTF-16 data to obtain character numbers, which
   are then encoded in UTF-8 as described above.  This contrasts with
   CESU-8 [CESU-8], which is a UTF-8-like encoding that is not meant for
   use on the Internet.  CESU-8 operates similarly to UTF-8 but encodes
   the UTF-16 code values (16-bit quantities) instead of the character
   number (code point).  This leads to different results for character
   numbers above 0xFFFF; the CESU-8 encoding of those characters is NOT
   valid UTF-8.

The Unicode definition today is the same, though in the past it's been
more wishy-washy. (CESU-8, BTW, is the official name for Oracle's and
MySQL's "UTF-8" encoding for database strings: the real thing is called
"AL32UTF8" and "utf8mb4" in Oracle and MySQL respectively.)

> This will break lots of things, not just UTF-8 decoders (most of which,
> I bet, will never actually notice).  -T

Modern ones that pay attention to spoofing most definitely will.

-- 
John Cowan                                   [email-masked]
        "You need a change: try Canada"  "You need a change: try China"
                --fortune cookies opened by a couple that I know

---

From: "Manger, James H" <[email-masked]>
Date: Mon, 17 Jun 2013 15:02:36 +1000

> >The point is that if JSON is encoded in UTF-8, any surrogate code
> >points MUST be escaped, even though the grammar does not say so.
> 
> unescaped = %x20-21 / %x23-5B / %x5D-%xD7FF / %E000-%10FFFF  ; Any
> unicode code point except control characters, QUOTATION MARK,  ;
> REVERSE SOLIDUS, or code points reserved for UTF-16 surrogates

+1
Unpaired surrogates cannot be interchanged reliably so they should be dropped from this ABNF. I don't mind a note saying how some implementations handle them (or their escaped form).

Fixing typos and tweaking the comment:

  unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF
    ; any Unicode scalar value, except those that must be escaped
    ; (control characters, quotation mark, and reverse solidus)

--
James Manger

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 17 Jun 2013 06:59:07 -0700

>    ; (control characters, quotation mark, and reverse solidus)

<no hat>

Making comments in ABNF that disagree with the ABNF itself seems like a completely terrible idea that will lead to lack of interoperability. Complicated ABNF is better than simple ABNF that has contradictory comments.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Mon, 17 Jun 2013 10:14:09 -0400

Paul Hoffman scripsit:

> Making comments in ABNF that disagree with the ABNF itself
> seems like a completely terrible idea that will lead to lack of
> interoperability. Complicated ABNF is better than simple ABNF that
> has contradictory comments.

Where's the contradiction?  The prose says exactly what the ABNF says.
%22 (quotation mark), %5C (backslash), and %D800 through %DFFF (surrogate
pairs) are left out of the full Unicode range.

-- 
No saves, Antonio, loke es morirse en su lingua. Es komo            John Cowan
kedarse soliko en el silensyo kada dya ke Dyo da, komo          [email-masked]
ser sikileoso sin saver porke.                      http://www.ccil.org/~cowan
                        --Marcel Cohen, 1985

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 17 Jun 2013 07:23:00 -0700

> pairs) are left out of the full Unicode range.

U+0000-0019 is not the entire list of Unicode control characters? U+0022 is not the only Unicode quotation mark.

Did we not learn anything from the IDN WG work 13 years ago?

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Mon, 17 Jun 2013 10:32:32 -0400

> is not the only Unicode quotation mark.

I assume by "19" you mean "1F".  If you want to say "C0 control
characters", fine.  But there is only one Unicode character named
"QUOTATION MARK", and it is the only one relevant to JSON.

> Did we not learn anything from the IDN WG work 13 years ago?

I don't see what that has to do with the price of pídàn in China.
Other quotation marks and control characters need not be escaped in JSON.

-- 
John Cowan  [email-masked]   http://ccil.org/~cowan
"The exception proves the rule."  Dimbulbs think: "Your counterexample proves
my theory."  Latin students think "'Probat' means 'tests': the exception puts
the rule to the proof."  But legal historians know it means "Evidence for an
exception is evidence of the existence of a rule in cases not excepted from."

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 17 Jun 2013 07:43:26 -0700

On Jun 17, 2013, at 7:32 AM, John Cowan <[email-masked]> wrote:

> I assume by "19" you mean "1F".  

Yes. :-)

> If you want to say "C0 control
> characters", fine.  

That would work for me. But I suspect that some people will say "no, all control characters".

> But there is only one Unicode character named

Then we should say that. Or, better, not have a comment that is open to misinterpretation.

> I don't see what that has to do with the price of pídàn in China.

An excellent accidental analogy! I was referring to the fact that "everyone" understood that the "dot" between DNS labels was U+002E... except the Japanese. And we had to put a complicated exception in for them, late in the process.

> Other quotation marks and control characters need not be escaped in JSON.

If everyone agrees with that, then we do not need the comment at all. But I suspect that "everyone" does not.

--Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 17 Jun 2013 17:21:41 +0000

>    ; (control characters, quotation mark, and reverse solidus)

After discussion, this?

unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF
 ; any Unicode scalar value, except those that must be escaped
 ; (C0 control characters,
 ;  "QUOTATION MARK" [U+0022], and
 ;  "REVERSE SOLIDUS" [U+005C])

The caps and codepoint references make it clear we're only talking about
those precise codepoints.

Others may argue that non-C0 control characters should be prohibited, but
that's a separate backward-compatibility argument from this starting point.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 17 Jun 2013 10:27:43 -0700

> that's a separate backward-compatibility argument from this starting point.

What value is that comment? Doesn't it invite misinterpretation of the ABNF, which is exact?

--Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 17 Jun 2013 17:30:13 +0000

>What value is that comment? Doesn't it invite misinterpretation of the

We have comments after most everything else.  Some of the readers of this
doc aren't going to be savvy enough to be able to read the ABNF easily
without the comment.

However, I'd be fine with removing it.

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Jun 2013 19:56:38 +0200

On Jun 17, 2013, at 19:27, Paul Hoffman <[email-masked]> wrote:

> What value is that comment?

Well, comments are always evil, because they mislead from the code.

> Doesn't it invite misinterpretation of the ABNF, which is exact?

The same argument could be made for

      begin-array     = ws %x5B ws  ; [ left square bracket

Actually, I'd argue for keeping in that style.

(And I know about U+27E6, U+301A, even U+FF3B.)

If we do want to explain, do it like this?

unescaped = ; any Unicode scalar value, except those that must be escaped:
          ; must escape C0 control characters %x0-1F
            %x20-21
          ; must escape %x22 " quotation mark
          / %x23-5B
          ; must escape %x5E \ reverse solidus
          / %x5D-D7FF
          ; not Unicode scalar values: %xD800-DFFF
          / %xE000-10FFFF

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 17 Jun 2013 11:23:46 -0700

Even though there are currently comments for all the simple strings, my personal preference would not to be to add a comment to this more complex one, and to instead let the ABNF stand on its own. 

So far, the proposed comment additions have confused what JSON requires to be *escaped* from what JSON *allows as codepoints* in strings. Adding comments to spell out the character names will make this misunderstanding worse, in my opinion.

--Paul Hoffman

---

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 18 Jun 2013 00:19:03 -0700

On Jun 12, 2013, at 15:54 , Tim Bray wrote:

> On Wed, Jun 12, 2013 at 3:46 PM, Norbert Lindenberg <[email-masked]> wrote:
> We’re not allowed to change JSON but we are allowed to improve the spec. The term “solidus” is obscure and off-putting.

If that's how you feel: Is there a proposal then to replace all occurrences of "solidus" and "reverse solidus" in the RFC? There are more, and the issue doesn't seem connected to the character vs. code point issue.
 
>> These escape sequences aren't about 16-bit quantities - they represent Unicode BMP code points.
>  
> They also represent surrogates.  Thus they combine fish and bicycles, and I think the only accurate way to describe what you can escape is “16-bit quantities”

Surrogate code points are BMP code points (see http://www.unicode.org/glossary). They are an awkward kind of code point and are best avoided, but I don't see what fish or bicycles have to do with them.

Also, as I explained before, in some environments there are no 16-bit quantities in either input or output.

Norbert

---

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 18 Jun 2013 00:43:56 -0700

> Modern ones that pay attention to spoofing most definitely will.

I did a bit of testing: In the current versions of the major desktop browsers (Firefox, Chrome, Safari, Explorer, Opera) the UTF-8 decoders used for external scripts and XMLHttpRequest replace CESU-8 encoded surrogate code points with replacement character sequences of varying lengths. In the current version of Node.js, the UTF-8 decoder used for external scripts will pass them through, with the effect that a sequence of CESU-8 encoded surrogate code points can turn into a valid supplementary character in UTF-16.

Norbert