Subject: [Json] Questions regarding the text model in 4627

From: Pete Resnick <[email-masked]>
Date: Mon, 15 Jul 2013 21:58:18 -0500

This is not *specifically* about the issue of json strings, though the 
answers here will certainly inform that discussion.

What do you believe is the theoretical model of 4627 regarding the 
characters that make up JSON objects? The question is not the original 
design intention of the authors, but rather about how you see the model 
in how you (or others) are using the document. So:

     a) Is it a stream of theoretical characters interpreted as JSON 
objects? Or:

     b) Is it a stream of integers, interpreted as characters, and then 
interpreted as JSON objects?
         b') If it is a stream of integers, is there an upper limit on 
the value of those integers in a JSON object? (Interesting choices 
include 255, 65535, 4294967295. 32767 would be an example of an 
*extremely* interesting choice.) Or:

     c) Is it a stream of (8-bit/16-bit-/32-bit) words, interpreted as 
characters, and then interpreted as JSON objects?

If you don't understand the distinction between any of the above 
choices, that's also an interesting piece of information.

I look forward to your answers.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 16 Jul 2013 16:48:55 +0200

>in how you (or others) are using the document. So:

The discussion in the Working Group has been aroung the data model for
string literals, while the characters that make up JSON objects are `{`
and `}` primarily. I believe there is rough consensus in the group to
define that unpaired surrogate escapes in string literals do not make
JSON documents malformed and non-conforming and point them out as an
interoperability problem along with common and good coping strategies.

>characters, and then interpreted as JSON objects?

At one level JSON documents are made of Unicode scalar values, that is
what goes into encode_utf8(...) and what comes out of decode_utf8(...).
They are often thought of as integers and as characters but that can be
misleading at times; for instance, the Unicode specification calls some
of the Unicode scalar values "non-characters" but often people include
those when they say "characters". There is an uppler limit of U+10FFFF.

In other words, the textual JSON syntax is a formal language over an
alphabet that is a proper subset of the set of Unicode scalar values.
That alphabet excludes integers like 0x0000 and 0xd800. I think that
is all well-understood and undisputed in the Working Group.

As an aside:

There is one thing your message helped me appreciate better, in ecma-
script `JSON.parse("\"\ud800\"") == JSON.parse("\"\\ud800\"")` is true
and that is not quite in keeping with my answers above, but the domain
of JSON.parse being a superset of the domain of encode_utf8(...) is a
point too subtle to address in the specification.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Tue, 16 Jul 2013 08:56:23 -0700

On Mon, Jul 15, 2013 at 7:58 PM, Pete Resnick <[email-masked]>wrote:

> design intention of the authors, but rather about how you see the model in
> how you (or others) are using the document. So:
>

I believe the design intent of 4627 is that strings be used to contain, as
it says explicitly and clearly in the introduction, sequences of Unicode
characters.    This is how it is universally interpreted by the large
community of people who use JSON for network APIs and assume that the only
useful purpose of strings is to contain Unicode text, and any instances of
bit patterns that cannot be interpreted as ordinary unicode characters is
simply an error.

>     a) Is it a stream of theoretical characters interpreted as JSON

It should be a stream of Unicode characters.

>         b') If it is a stream of integers, is there an upper limit on the
> value of those integers in a JSON object? (Interesting choices include 255,
> 65535, 4294967295. 32767 would be an example of an *extremely* interesting
> choice.) Or:
>

I wouldn’t worry about the use of unicode codepoints that could be
characters but haven't been assigned yet.  The chance of this happening by
accident is low, and ignoring it future-proofs us against future Unicode
releases.

> characters, and then interpreted as JSON objects?

No, it is what the introduction of the RFC says it is, a sequence of
Unicode characters. -T

>
> If you don't understand the distinction between any of the above choices,
> that's also an interesting piece of information.
>
> Pete Resnick<http://www.qualcomm.**com/~presnick/<http://www.qualcomm.com/~presnick/>
> >
> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: Tim Bray <[email-masked]>
Date: Tue, 16 Jul 2013 09:24:25 -0700

OK, I guess I misinterpreted Pete's question. Let me try again.

On Mon, Jul 15, 2013 at 7:58 PM, Pete Resnick <[email-masked]>wrote:

> characters that make up JSON objects?

Yes.  At greater length:

The model I see is that the bit patterns in JSON texts exist to represent
sequences of Unicode characters.  There are a variety of ways of doing this
(UTF-8/16/32), but they encode integers which are, in the case of UTF-8 and
UTF-32, Unicode code points, and in the case of UTF-16, code points, and
surrogate values which are then used to construct Unicode code points.
When parsing JSON texts, all the “syntax” characters ([{},":\] and
whitespace) are ordinary unicode characters, so lexing/parsing into JSON
objects, arrays, numbers, booleans, and strings is straightforward.

Inside names and strings, I regard any sequence of bit patterns which are
not readable as sequences of Unicode code points which represent characters
as an error condition.

---

From: Pete Resnick <[email-masked]>
Date: Tue, 16 Jul 2013 11:25:49 -0500

On 7/15/13 9:58 PM, Pete Resnick wrote:
> design intention of the authors, but rather about how you see the 
> model in how you (or others) are using the document. So:
>

It's clear from Bjoern and Tim's answers that I was not clear. I thought 
I was clear, but they didn't answer the questions I asked at all. So let 
me try to be clearer:

1. The questions are not about strings in JSON. If you answer about 
strings, you haven't understood the questions. The questions are about 
the JSON object itself. I understand that strings appear in JSON 
objects. But I am not asking about something that is special about 
strings as against the rest of the JSON object. I am asking about the 
JSON object.

2. I am not asking about the design intention of 4627. If you tell me 
what you think the design intention of 4627 is, you haven't understood 
the questions. I am asking about your view of the theoretical model.

3. The questions are not about Unicode codepoints. You will not see the 
words "Unicode" or "codepoints" in the questions.

4. The questions are not looking for your opinion of what others think. 
They ask what *your* answers to the questions are. Resist the temptation 
to talk about other people's opinions. What are your thoughts?

These are very theoretical questions. The answers are not directly going 
to tell us what text will go into an Internet draft. But I do think this 
will help get us to something useful.

I again look forward to your answers.

pr

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Jul 2013 11:43:20 -0500

On Mon, Jul 15, 2013 at 9:58 PM, Pete Resnick <[email-masked]> wrote:
>     a) Is it a stream of theoretical characters interpreted as JSON objects?

What does "theoretical characters" mean?

> Or:

A JSON document is a textual object, specifically one that can be
presented to humans safely (e.g., no binary non-textual data must
appear in the encoded form).

The text in question should be Unicode text.  A UTF for JSON documents
is not required, and one could use any UTF, or even any other codeset,
provided that data is not lost.  (E.g., if you escape all JSON string
contents that can't be represented in EBCDIC, then output a JSON
document in EBCDIC, then you should be able to parse it back into a
faithful internal representation of the darned thing.)

> 65535, 4294967295. 32767 would be an example of an *extremely* interesting

It's Unicode data.  In any UTF the highest code point value would be
0x10FFFF.  As to what appears on the wire/disk/screens/whatever, that
depends on the UTF chosen.  UTF-8 is a sequence of bytes, so the
highest valued byte that can appear would be... what, 0xF4?  Perhaps
I'm misunderstanding the question.

> characters, and then interpreted as JSON objects?

Yes, see above.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Tue, 16 Jul 2013 13:57:37 -0400

Pete Resnick scripsit:

> characters that make up JSON objects? The question is not the
> original design intention of the authors, but rather about how you
> see the model in how you (or others) are using the document. So:
> 
>     a) Is it a stream of theoretical characters interpreted as
> JSON objects? Or:
> 
>     b) Is it a stream of integers, interpreted as characters, and
> then interpreted as JSON objects?
>         b') If it is a stream of integers, is there an upper limit
> on the value of those integers in a JSON object? (Interesting
> choices include 255, 65535, 4294967295. 32767 would be an example
> of an *extremely* interesting choice.) Or:
> 
>     c) Is it a stream of (8-bit/16-bit-/32-bit) words, interpreted
> as characters, and then interpreted as JSON objects?
> 

Your question is ill-posed in at least two respects, and so people who
have certain beliefs are trying to answer the question they think you
meant to ask.  (See "Geek Answer Syndrome".)

1) Your question does not make clear what "object" means.  Do you mean
whatever in JSON is not an array, string, number, or boolean?  Or do
you mean an entity body which purports to conform to RFC 4627?

2) Your question contains a presupposition, namely that such "objects"
or entity bodies are composed of a stream of characters.  In the case
of objects-as-defined-in-the-RFC, I certainlly don't believe that and I
don't think anyone does. In the case of entity bodies, I do believe it,
but others may not.

But taking your question at face value, my reply is the conjunction of
(d1) and (d2) below:

(d1) An "object", as defined in the JSON spec, is not a stream of
characters.

(d2) A JSON entity body has nothing to say about the characters which do
indeed (so I believe) compose it.  "Character" is an undefined term in RFC
4627, which dictates only what characters may appear and in what order.

> I look forward to your answers.

I look forward to a rephrasing of your question to eliminate the
problems above.

-- 
BALIN FUNDINUL          UZBAD KHAZADDUMU        [email-masked]
BALIN SON OF FUNDIN     LORD OF KHAZAD-DUM      http://www.ccil.org/~cowan

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 16 Jul 2013 19:58:23 +0200

>I was clear, but they didn't answer the questions I asked at all. So let 

None of the choices comes close to how I see the model and I explained
how I see it instead. The problem starts with your use of "JSON object".
I see JSON objects as a data model concept, they are collections of
pairs. That is how RFC 4627 defines the term "object". They aren't made
up of characters. RFC 4627 defines how objects and arrays and other
values are represented in textual form, but the options are all for the
reverse. I try to think more abstractly than serial processes. There is
no difference between `[1,2,3]` and the output of encode_utf8("[1,2,3]")
to me as far as any model is concerned, so I would not put my name be-
hind the "and then" options b) and c) for that reason alone.

Your clarifications are just repetitions of what you wrote or implied,
they are not adding new information that could help giving answers.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 16 Jul 2013 11:09:27 -0700

On Tue, Jul 16, 2013 at 9:25 AM, Pete Resnick <[email-masked]>wrote:

> It's clear from Bjoern and Tim's answers that I was not clear. I thought I
> was clear, but they didn't answer the questions I asked at all. So let me
> try to be clearer:
>
>
You have managed to rule all answers to all question you asked so far to be
invalid. The only thing missing was actual letter grade for how close any
of us were.
Perhaps you should answer your own questions so that they satisfy criteria
you have for answers.

-+ Tatu +-

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 16 Jul 2013 18:13:34 +0000

Again, replying before reading others' replies.

>*extremely* interesting choice.) Or:

I think this is the closest.  The semantic model is that there is a stream
of integers, each of which is less than 10FFFFh (1114111d), and is
interpreted as a Unicode Scalar Value.  That semantic may be rendered into
octets on the wire using one of 5 encodings (UTF-8, UTF-16BE, UTF-16LE,
UTF-32BE, UTF-32LE), of which UTF-8 is both preferred and the default.
Inside of a JSON string (*char in the ABNF), aside from certain
exceptions, these integers are intended to be rendered into appropriate
objects in the local programming environment that typically have names
like "Character", "char", etc.  Of the exceptions, there are two types.
Named escapes, such as \n, represent specific Unicode Scalar Values, and
will usually be rendered into the local programming environment as a
Character that represents that scalar value.  Numeric escapes, such as
\u005c, represent 16-bit code units that are intended to be interpreted
similarly to UTF-16.  When rendered in the local programming environment,
they are to be hex-decoded, then all adjacent numeric escapes are to be
UTF-16 decoded together, and the resulting Unicode Scalar Values are
usually rendered to Characters as above.

Expected behavior when errors in decoding are detected at either the first
level (UTF-8 to integers) or the second (\u escapes to UTF-16 to integers)
are not currently specified.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Tue, 16 Jul 2013 14:17:54 -0400

> characters that make up JSON objects? 

Okay, having answered your question in my previous post, I am now
going to ask and answer the question I believe you should have asked.
If this be Geek Answer Syndrome, make the most of it.

# What do you believe is the theoretical model of 4627 regarding the
# elementary units of which JSON texts are composed?

I should have said "texts" rather than "entity bodies" in my previous
post, as "JSON text" is the term used by the RFC.

> objects? 

Almost.  I would reword this as follows:

# Is it a sequence of abstract characters (see Unicode 6.2, section 3.4,
# definition D7) interpreted as a JSON object or array?

To which my answer is unequivocally "Yes".

A JSON text is a sequence of characters.  It may be encoded as a
sequence of integers (which may be encoded as a sequence of bytes).
It may also be encoded by a sequence of atoms, as when a JSON text is
printed on a piece of paper.  It may even be encoded by the *absence*
of certain atoms, as when a JSON text is carved in stone.  It may not be
encoded at all (as far as we know), as when I only *imagine* a JSON text.
But in all these cases, the text is a sequence of characters.

> choices include 255, 65535, 4294967295. 32767 would be an example of
> an *extremely* interesting choice.) Or:

No.  A text may be encoded as a stream of integers, but also in other
ways.

> as characters, and then interpreted as JSON objects?

No.  A text may be encoded as a stream of 8-bit, 16-bit, or 32-bit words,
but also in other ways.

> choices, that's also an interesting piece of information.

I believe I do understand the distinctions.

-- 
De plichten van een docent zijn divers,         John Cowan
die van het gehoor ook.                         [email-masked]
      --Edsger Dijkstra                         http://www.ccil.org/~cowan

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 16 Jul 2013 18:25:09 +0000

>But in all these cases, the text is a sequence of characters.

OK, I like this better than integers for the overall conceptual model.  I
think that's the right semantic, but in practice the next step in the
programming of JSON is to start to think about Unicode Scalar Values in
their integer embodiments.

-- 
Joe Hildebrand

---

From: Pete Resnick <[email-masked]>
Date: Tue, 16 Jul 2013 15:30:57 -0500

Excellent. The latest messages, especially in this thread, are 
generating the conversation I was looking for. I already see some very 
interesting (and in a couple of cases, very surprising) comments. I 
start with John's message simply because it had a particularly 
interesting meta-bit that I thought it worth commenting about, but other 
messages are very useful as well. And I see some serious disagreements 
in some of the responses so far, and I think resolving those (or at 
least understanding why people have different models) will be useful. 
All good.

So, in particular:

> meant to ask.  (See "Geek Answer Syndrome".)

I think the below are fair criticisms, though in some sense I did want 
to leave some concepts loose enough to see where people went with them 
so that we could come up with good common terminology. But to speak to each:

> whatever in JSON is not an array, string, number, or boolean? Or do you mean an entity body which purports to conform to RFC 4627?
>    

In your subsequent message, you refer to "JSON text". I think that's 
pretty closer to what I intended, though I didn't want to 
over-constrain. But using "object" almost certainly over-constrains too, 
since it refers to one of the container types, but also might refer -- 
at least in my head -- to the generalized concept of a "JSON thingy". I 
certainly didn't mean "whatever in JSON is not an array, string, number, 
or boolean".

So, take an email message for example. An email message can contain 
words and pictures and sounds and other theoretical objects. But for 
purposes of layer 7, and in the context of (for example) and RFC 
822/5322 message, an Internet Mail Format message is a collection of 
text, and is a series of characters. Words and pictures and sounds can 
be represented (one layer up), and the entire message can be encoded 
(one layer down), but at the level of description of the spec, an email 
message *is* characters. I'm trying to get a grip on how people view 
JSON "thingys" (just to make up a new term) on the same level of 
description.

> don't think anyone does. In the case of entity bodies, I do believe it,

So, do you take "objects" and "arrays" and other things inside a JSON 
thingy to be analogous to pictures and words and sounds in an email 
message, and therefore not be "composed of characters", but the overall 
JSON thingy *is* composed of characters?

> 4627, which dictates only what characters may appear and in what order.

Reasonable answers.

pr

---

From: John Cowan <[email-masked]>
Date: Tue, 16 Jul 2013 18:05:43 -0400

Pete Resnick scripsit:

> So, do you take "objects" and "arrays" and other things inside a
> JSON thingy to be analogous to pictures and words and sounds in an
> email message, and therefore not be "composed of characters", but
> the overall JSON thingy *is* composed of characters?

Yes.  I think it would be nice if JSON strings (unlike objects, arrays,
booleans, and numbers) were composed of characters, but unfortunately
I don't think they are: they are composed of UTF-16 code units.

-- 
May the hair on your toes never fall out!       John Cowan
        --Thorin Oakenshield (to Bilbo)         [email-masked]

---

From: "Jim Schaad" <[email-masked]>
Date: Tue, 16 Jul 2013 18:13:24 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Pete Resnick
> Sent: Monday, July 15, 2013 7:58 PM
> To: [email-masked]
> Subject: [Json] Questions regarding the text model in 4627
> 
answers
> here will certainly inform that discussion.
> 
design
> intention of the authors, but rather about how you see the model in how
you
> (or others) are using the document. So:
> 
>      b) Is it a stream of integers, interpreted as characters, and then
interpreted
> as JSON objects?
value of
> those integers in a JSON object? (Interesting choices include 255, 65535,
> 4294967295. 32767 would be an example of an
characters,
> and then interpreted as JSON objects?

Specifically 8-bit octets.  This is because I care about something that can
be fed in to an encryption function and also converted into an in memory
form  that holds structure.

Jim

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 16 Jul 2013 19:14:57 -0700

On Jul 15, 2013, at 19:58 , Pete Resnick <[email-masked]> wrote:

> What do you believe is the theoretical model of 4627 regarding the characters that make up JSON objects?

As discussed in the meantime, I read "JSON object" as "JSON text".

>  c) Is it a stream of (8-bit/16-bit-/32-bit) words, interpreted as characters, and then interpreted as JSON objects?

Close to c), although instead of the undefined term "characters" it should say "Unicode code points" or "Unicode scalar values", depending on the environment in which JSON is used. Most of the RFC is only concerned with the stream of code points; the code units come into play in sections 3 and 6.

Norbert

---

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 16 Jul 2013 19:22:20 -0700

> But in all these cases, the text is a sequence of characters.

I'd rather not go there. If we include JSON text printed in paper or carved into stone into the scope of the RFC, then our interoperability discussion will have to include the coverage of the Unicode character set of available fonts, the print resolution, the quality of OCR algorithms when applied to the Unicode character set, handling of user-defined or not-yet-defined characters, Unicode normalization, and probably a few additional factors. Compared to the data loss and data corruption users would have to expect with these generators and parsers, the limitations on number size or surrogate code points discussed so far would be trivial.

Norbert

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 17 Jul 2013 06:47:12 +0000

>I don't think they are: they are composed of UTF-16 code units.

Can you say more about this?  I'm not seeing it yet.

If I send you an octet stream that is identified (e.g. via an HTTP
Content-Type header) as application/json that consists of the following
octets (hex-encoded here for crystal clarity):

5b 22 f0 90 85 83 22 5d

You can tell using section 3 that this is encoded using UTF-8.  In my
view, you have to UTF-8 decode before using the grammar to parse.  If the
parser generates strings in your programming language that are natively
UTF-32, you would end up with an array containing one element that is a
string consisting of one UTF-32 code unit.  I don't see where there was
ever a UTF-16 code point involved.

If the parser generates JavaScript-style strings that are UTF-16 code
units, then that is an implementation choice, not a property of the
underlying semantic model.

-- 
Joe Hildebrand

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 17 Jul 2013 16:31:41 +0900

[independent reply, before reading other's replies]

[Excludyng cases such as JSON printed out on paper or written on napkins.]

I'd say that JSON is a stream of 8-bit/16-bit-/32-bit words, interpreted 
as a stream of integers, interpreted as a stream of characters.

That's a combination of b) and c), which includes a). That's what you 
end up when carefully analyzing how characters are represented on 
computers with a generality that can deal with the range of phenomena 
found in character encoding. That's what all the relevant documents from 
the IETF, W3C, and Unicode explain (although they may differ in details 
or where they put the focus).

However, to most intents and purposes, that's not what JSON should be 
concerned with, because that's common to all textual formats. It's just 
implicit when we say that UTF-8 is used for transfer over the Internet, 
or something along these lines.

Regards,   Martin.

On 2013/07/16 11:58, Pete Resnick wrote:
> b') If it is a stream of integers, is there an upper limit on the value
> of those integers in a JSON object? (Interesting choices include 255,
> 65535, 4294967295. 32767 would be an example of an *extremely*
> interesting choice.) Or:
>

---

From: John Cowan <[email-masked]>
Date: Wed, 17 Jul 2013 08:13:30 -0400

Joe Hildebrand (jhildebr) scripsit:

> string consisting of one UTF-32 code unit.  

The trouble comes in when your parser receives these octets:

5b 22 5c 75 44 38 30 30 22 5d

What should it do?  It could report an implementation restriction, or
it could return invalid UTF-32.  Because unfortunately there's no doubt
that this is valid JSON.

-- 
Babies are born as a result of the              John Cowan
mating between men and women, and most          http://www.ccil.org/~cowan
men and women enjoy mating.                     [email-masked]
    --Isaac Asimov in Earth: Our Crowded Spaceship

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 17 Jul 2013 18:10:00 +0000

>it could return invalid UTF-32.  Because unfortunately there's no doubt

In my worldview, this is an ill-formed escape sequence.  Unicode 6.2
section 3.9, D75 says: "Isolated surrogate code units have no
interpretation on their own", which seems pretty clear.

I see the escape mechanism as being explicitly for "characters" (where I
see "Unicode Scalar Value" as a more precise word for what the author
likely intended), where those "characters" might be either from the BMP or
not.  The escaping mechanism is not for 16-bit code units, and I think
that is made clear in the text of section 2.5, even if the ABNF has a bug
that doesn't match the text.

The parser might make one of several choices when it encounters an
isolated surrogate code unit escape:

- Error (useful for security-relevant things)
- Replace with U+FFFD (only useful for strings intended for human
consumption)
- Maintain the data in a form that, while not valid Unicode, is still
useful in some unspecified way (what ECMAscript does)

Note: I'm trying to describe what my mental model is, not saying that
everyone else should think this way.

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Wed, 17 Jul 2013 11:28:45 -0700

My model is entirely consistent with Joe’s. -T

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 17 Jul 2013 19:21:16 +0000

That model could encoded in ABNF that looks like this:

      char = unescaped / named-escape / bmp-escape / extended-escape
      unescaped = %x20-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF
      named-escape =
          escape (
              %x22 /          ; "    quotation mark  U+0022
              %x5C /          ; \    reverse solidus U+005C
              %x2F /          ; /    solidus         U+002F
              %x62 /          ; b    backspace       U+0008
              %x66 /          ; f    form feed       U+000C
              %x6E /          ; n    line feed       U+000A
              %x72 /          ; r    carriage return U+000D
              %x74 )          ; t    tab             U+0009
      escape = %x5C              ; \
      bmp-escape = escape %x75 (
              not-d-hex 3HEXDIG /
              "D" %x30-37 2HEXDIG
              )
      not-d-hex = DIGIT / "A" / "B" / "C" / "E" / "F" ; hex, except for D
      extended-escape =  high-surrogate low-surrogate
      high-surrogate = escape %x75 "D" high-hex 2HEXDIG
      high-hex = %x38-39 / "A" / "B"
      low-surrogate = escape %x75 "D" low-hex 2HEXDIG
      low-hex = "C" / "D" / "E" / "F"

which is of course ugly, but absolutely explicit.  I believe that the
suggested error handling / recovery patterns are independent from the ABNF.

On 7/17/13 12:28 PM, "Tim Bray" <[email-masked]> wrote:

>My model is entirely consistent with Joe¹s. -T
>
>
>
>On Wed, Jul 17, 2013 at 11:10 AM, Joe Hildebrand (jhildebr)
><[email-masked]> wrote:
>

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Jul 2013 14:42:06 -0500

> interpretation on their own", which seems pretty clear.

I agree as to unescaped code units.

> I see the escape mechanism as being explicitly for "characters" (where I

That's not clearer than anything else about JSON.  It'd be entirely
reasonable for the ECMAScript crowd to say "well, unpaired surrogates
have to be sent as escaped code units, but obviously not as
unescaped".  That is, two sides of the argument would disagree over
whether a string could contain unpaired surrogates yet agree that the
JSON *text* containing such a string MUST NOT contain unpaired
surrogates.

> that doesn't match the text.

A literal reading of section 2.5 is that strings are of Unicode
characters.  And yet a large subset of the JSON ecosystem (ECMAScript
and friends) doesn't agree.

> - Replace with U+FFFD (only useful for strings intended for human

OK, so this brings up questions that I've asked before:

1) Are escaped <thing>s supposed to be equivalent to their unescaped
forms, for string and deep JSON text comparison purposes?

2) Must/should parsers that can natively represent a given escaped
<thing> unescape it in the results in produces?

3) If an escaped <thing> appears in a JSON string that would result in
invalid UTF-8/16/32 when unescaped, what should the parser do?

RFC4627 says nothing about this.  Much of the discussion seems to
assume (or makes most sense to *me*when I assume that the commenters
assume) that the answers to (1) and (2) are "yes".

My answers would be "yes" to (1), "at the parser's/application's
option" for (2), and "optionally reject or pass through in escaped
form" for (3).

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 17 Jul 2013 12:49:03 -0700

> and friends) doesn't agree.

I admit that I don’t have a good feeling for the structure of the JSON
ecosystem. In the part of it that I can see (people who are using it to
structure messages in application-level APIs), the belief that strings are
for Unicode characters, and only for Unicode characters, and that anything
in there that isn’t a Unicode character is a bug, is pretty well 100%.

BTW, I agree with Nico on #1 and #2, and while I’m grumpy about #3, I
suspect our charter doesn’t allow us to force-classify that situation as an
error, which would be my preference. -T

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Jul 2013 15:12:55 -0500

> in there that isn’t a Unicode character is a bug, is pretty well 100%.

If the server side of your API is running in node.js (which is very
popular nowadays, so it well might be) then you might be dealing with
ECMAScript (since that's what node.js builds on, just like JavaScript
on the browser side).

> error, which would be my preference. -T

Were those useful questions then?!  I mean, if you and Joe and others
can agree with my answers to those three questions then we might well
be able to reach a consensus as to strings that we can all live with.

I understand the grumpiness re: #3, but it seems like a reasonable
compromise (and yes, a bit more code for parsers and generators, but
only a tiny bit).

I once implemented a convention for encoding binary strings as an
object with a specific URN as the only name and base64-encoded string
as the value, with the parser and generator both implementing this
convention  My answer to my #3 is not too unlike such a convention,
but with the benefit that the \u escapes are already in the spec *and*
some implementations already handle binary strings via \u escapes, so
\u escaping is a bit of a more convenient convention for non-Unicode
(e.g., octet) strings.

Note that my answer to my question #3 effectively allows JSON
generators and parsers that natively support binary strings to
generate a binary string by escaping all 16-bit code units that cannot
form valid Unicode characters, and to detect binary strings on parsing
by noticing that some escaped values cannot be unescaped to produce
valid Unicode strings, therefore the input must be a binary string.
Note that some binary strings can be interpreted as valid Unicode
strings and so there's an ambiguity on the parser side (that the
application can resolve by casting if it cares about a given string
value).  Note too that this convention would require that \u00xx be
decoded as a single octet, otherwise there's a problem with odd-length
octet strings.

Such an encoding of binary data would make me somewhat happy: I hated
my {"urn:...":"<base64>"} convention (but base64 will often be more
efficient).  But this implication will surely upset you, so before you
decide that you can grumpily agree to my answer to my #3 you should
understand the  consequence described above.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 17 Jul 2013 20:33:17 +0000

On 7/17/13 1:42 PM, "Nico Williams" <[email-masked]> wrote:

>That's not clearer than anything else about JSON.

Sure.  But that's my worldview.  There are several different models that
could be made to fit the language in 4627.

>It'd be entirely

Agree.  Note that the difference between "string MAY contain unpaired
surrogates" and "string MUST NOT contain unpaired surrogates, and parsers
MAY maintain as much state as possible when they detect a parse error" is
pretty slim.

>and friends) doesn't agree.

In the ECMAscript spec, they're talking about a series of UTF-16 code
units, because the JSON object assumes that you have already decoded the
original encoding (e.g. UTF-8) to an internal representation
(error-tolerant UTF-16), and are specifying parsing rules for that
internal representation.  They have a handy mechanism for dealing with
invalid escapes, which is that same error-tolerant UTF-16.

That doesn't make either of the error classes less an error, it just means
they have prioritized not losing data in those error conditions, and have
decided that not throwing an error is the correct Postelian approach.  The
fact that some folks take advantage of that error-handling tolerance to do
fancy tricks is an unintended outcome in several other existing
error-tolerant systems.

>forms, for string and deep JSON text comparison purposes?

In my model, yes.  What matters is reproducing the semantic that generated
the encoded form.

><thing> unescape it in the results in produces?

Unqualified yes.  If they don't, they can't round trip, assuming that they
escape going the other direction.  Compare:

wire -> internal -> wire
["\u0000"] -> [ "\000" ] -> [ "\u0000" ]
["\u0000"] -> [ "\u0000" ] -> [ "\\u0000" ]

>invalid UTF-8/16/32 when unescaped, what should the parser do?

To be clear, I think the only way this can happen is with invalid
surrogate sequences.  Obviously if there's an encoding error a layer down
with invalid UTF8, an error must be thrown or some sort of error-tolerance
mechanism must be invoked.  See:

https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences

for some ideas.  And I think that there are multiple different answers,
which is why sending such sequences should be avoided if you want to
interop with code you don't control.

>RFC4627 says nothing about this.

Agree.  We could argue over what it implies, but not that it specifies how
to deal with errors.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Jul 2013 16:17:22 -0500

> MAY maintain as much state as possible when they detect a parse error" is

Hmm, well, it is, but maintainers of generators that will produce them
will object.

> invalid escapes, which is that same error-tolerant UTF-16.

The UTF-16 in ECMAScript is incidental.  What's fundamental is that
they allow *any* 16-bit code units, certainly escaped, and who knows
if unescaped (it'd be nice to find out).

> fancy tricks is an unintended outcome in several other existing

That's a reasonable view of the parser side.  But from the ECMAScript
pov it's probably not a reasonable view of the generator side.  Still,
as long as the parser can get by then the stricture on generators can
be disregarded safely (as long as crap is escaped), and we might have
consensus!

> In my model, yes.  What matters is reproducing the semantic that generated

I think we can have broad agreement on this.

> escape going the other direction.  Compare:

What if the <thing>, unescaped, results in invalid UTF/Unicode?
That's where #3 goes.

> ["\u0000"] -> [ "\u0000" ] -> [ "\\u0000" ]

Yes, but if my internal format is C strings... (I know, that means I
have other problems), or if the escaped code units included unpaired
surrogates and my native format must be valid UTF-8 (or 16, or 32),
then what?  I think the only sensible answers are:

 - reject
or
 - preserve the offending escapes in escaped form

> mechanism must be invoked.  See:

It could happen with \u0000 if my native representation is C strings.

> interop with code you don't control.

OK, but what's your answer to #3?  Is it determined by your
"unqualified yes" to #2?  I was hoping your answer to #3 would be to
allow the parser to preserve the offending escapes as escapes at the
application's or implementor's option (as well as to reject the input,
and perhaps map out the offending escapes).

> Agree.  We could argue over what it implies, but not that it specifies how

But I think we should state this equivalence explicitly.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 17 Jul 2013 14:34:55 -0700

On Wed, Jul 17, 2013 at 2:17 PM, Nico Williams <[email-masked]>wrote:

> pov it's probably not a reasonable view

There’s a community that apparently matters but which we’re not hearing
from. You keep talking about the ECMAscript needs and the ECMAScript POV.
This implies that there is a community of implementers whose success
depends on being able to stuff naked surrogates into JSON strings.  Could
we meet one, please?  I guess they’d have to be building  Browser-JS <=>
NodeJS apps, and that is a reasonably popular configuration, so it
shouldn’t be that hard to find someone. Could we actually have one such
person step up and explain their needs?  Rob Sayre, this sort of sounds
like your territory, care to chip in?

Because I’ve been using JSON all the time on a wide variety of apps for
like 5 years now and I’ve never met anyone who regarded payloads containing
non-characters as anything but an egregious bug.

I’m quite sure that there are groups to whose needs my experience has left
me oblivious. But this debate would work better if we could actually hear
from them what their needs are, and what we can't put in 4267bis without
breaking them. -T

> of the generator side.  Still,

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 17 Jul 2013 21:55:25 +0000

>Hmm, well, it is, but maintainers of generators that will produce them

I don't care too much.  Just because you can make something work doesn't
mean that you should.  Also, I bet they don't really care if they are
compliant with the letter of the law; their code will still work in
practice.

>if unescaped (it'd be nice to find out).

Not quite.  If I do feed ECMAScript a valid UTF-16 surrogate pair, it will
do interesting things with it.  For example:

console.log("\uD800\uDD43")

will print the glyph for GREEK ACROPHONIC ATTIC FIVE. I see my
characterization of their string type as fault-tolerant UTF-16 as
justified.  Also ECMA-262 section 4.3.16 says:

"""
NOTE A String value is a member of the String type. Each integer value in
the sequence usually represents a single 16-bit unit of UTF-16 text.
However, ECMAScript does not place any restrictions or requirements on the
values except that they must be 16-bit unsigned integers.
"""

JSON.parse takes a String.

>be disregarded safely (as long as crap is escaped), and we might have

That was what I was hoping for. Real-world interoperability while ensuring
that the moral indignation we feel toward those who generate technically
invalid output is captured in the spec.

>What if the <thing>, unescaped, results in invalid UTF/Unicode?

That's up to the parser.

Some may error, 
Some replace, 
Some may leave the trash in place.

No matter if they think it's trash
Parsers make a choice or crash

If you send half a pair
Please remember have a care
Parsers choose, you know not which,
How to handle surrogates

(sorry, it's been a long week already)

> - preserve the offending escapes in escaped form

Or convert to U+FFFD.

>It could happen with \u0000 if my native representation is C strings.

Yes, I see that.  The NULL would terminate the string.

>and perhaps map out the offending escapes).

I think the question isn't terribly accurate.  There are two pieces:

1) What should happen upon UTFx-decoding errors with the overall stream?

I have a strong bias for aborting with an error, but allow for the
possibility that folks may want to be tolerant of some classes of errors.
I think the spec should make a recommendation, but tell senders they MUST
NOT rely on specific behavior for interoperability.

2) What should happen upon UTF-16 decoding errors from unicode escape
sequences?

Effectively the same answer as above, but I have slightly more tolerance
for preserving the broken data, since there are several implementations
that do it today.  Regardless, senders MUST NOT rely on a specific
behavior for interoperability.

>But I think we should state this equivalence explicitly.

Yup.  I think that's a fine way forward.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Jul 2013 17:04:08 -0500

> from. You keep talking about the ECMAscript needs and the ECMAScript POV.

Er, I thought we'd heard from some people from that subset of the
community.  And there was even an odd exchange about who could and
could not represent ECMA itself here.

I myself am not a member of that community, so I can remind us that they exist.

There's also the issue that we may need a liason.  The WG's charter
explicitly mentions the ECMA.  But there may be no one representing
the ECMA's interests.  Can we make progress without a liason or
explicit call for participation from ECMA?

Nico
--

---

From: Jorge Chamorro <[email-masked]>
Date: Thu, 18 Jul 2013 02:45:40 +0200

> But in all these cases, the text is a sequence of characters.

And that's why we need -at least- *two* separate docs:

<http://www.ietf.org/mail-archive/web/json/current/msg00822.html>

On 13/06/2013, at 17:50, Douglas Crockford wrote:

> The confusion and controversy around this work is due to a mistake that I
> made in RFC 4627. The purpose of the RFC, which is clearly indicated
> in the title, was to establish a MIME type. I also gave a description of
> the JSON Data Interchange Format. My mistake was in conflating the two,
> putting details about the MIME type into the description of the format. My
> intention was to add clarity. That obviously was not the result.
> 
> JSON is just a format. It describes a syntax of brackets and commas that
> is useful in many contexts, profiles, and applications. JSON is agnostic
> about all of that stuff. JSON shouldn't even care about character encoding.
> Its only dependence on Unicode in the hex numbers used in the \u notation.
> JSON can be encoded in ASCII or EBCDIC or even Hollerith codes. JSON can
> be used in contexts where there is no character encoding at all, such as
> paper documents and marble monuments.
> 
> There are uses of JSON however in which such choices matter, and where
> behavior needs to be attached to or derived from the syntax. That is
> important stuff, and it belongs in different documents. Such documents
> will place necessary restrictions on JSON's potential. No such document
> can fit all applications, which causes much of the controversy we've seen
> here. One size cannot fit all. JSON the format is universal. But real
> applications require reasonable restrictions.
> 
> So we should be working on at least two documents, which is something we have
> discussed earlier. The first is The JSON Data Interchange Format, which is
> a simple grammar. The second is a best practices document, which recommends
> specific conventions of usage.

We're going in circles.
-- 
( Jorge )();

---

From: John Cowan <[email-masked]>
Date: Wed, 17 Jul 2013 21:08:17 -0400

> MAY maintain as much state as possible when they detect a parse error" is

Au contraire, it is fundamental.  If they MAY contain unpaired surrogates,
then it is legitimate to use JSON strings to encode sequences of
arbitrary 16-bit integers (with proper escaping).  If they MUST NOT,
then it is not legitimate to do so even if it happens to work for some
generator-parser combinations.

-- 
Even a refrigerator can conform to the XML      John Cowan
Infoset, as long as it has a door sticker       [email-masked]
saying "No information items inside".           http://www.ccil.org/~cowan
        --Eve Maler

---

From: R S <[email-masked]>
Date: Wed, 17 Jul 2013 18:10:43 -0700

> like your territory, care to chip in?

This message from Waldemar Horwat may be informative:

<https://mail.mozilla.org/pipermail/es5-discuss/2009-October/003385.html>

If I've learned anything about ECMAScript, it's this: if it works today,
there is unmaintained content depending on it. Breaking content is just
about the worst thing an ECMAScript implementation can do, so none of them
ever will in a market with 5+ credible implementations.

I'd recommend reading the entire thread I linked, particularly the messages
from Jason Orendorff and Douglas Crockford. Nothing has changed in the four
years since, which should tell this WG that we are not here to legislate
correctness, but to document the status quo.

- Rob

---

From: Tim Bray <[email-masked]>
Date: Wed, 17 Jul 2013 18:54:59 -0700

I read the 18 messages in the thread and the discussion was at the level of
theory. The constituency I’d like to hear from - actual developers who
actually exchange actual JSON strings that can’t be expressed as sequences
of Unicode codepoints - remains un-heard from.   Remember that rough
consensus & running code thing?  I’d really, really like to hear about
instances of running code that depend on behavior that flatly contradicts
the assertion in the RFC introduction that JSON strings are sequences of
Unicode characters.

Yes, I understand that:
- the ABNF allows arbitrary 16-bit quantities in strings, despite what the
intro says
- the ECMAScript standardization project has explicitly blessed this
- Douglas Crockford says receiving software should just shut up and deal
with naked surrogates

I just want to meet and understand the constituency we’re trying to serve
by allowing behavior that, to 100% of the users of JSON implementations
that I know, is self-evidently incorrect.  -T

On Wed, Jul 17, 2013 at 6:10 PM, R S <[email-masked]> wrote:

> I'd recommend reading the entire thread I linked, particularly the
> messages from Jason Orendorff and Douglas Crockford. Nothing has changed in
> the four years since, which should tell this WG that we are not here to
> legislate correctness, but to document the status quo.
>
> - Rob
>

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Jul 2013 21:48:37 -0500

On Wed, Jul 17, 2013 at 8:54 PM, Tim Bray <[email-masked]> wrote:
> I just want to meet and understand the constituency we’re trying to serve by
> allowing behavior that, to 100% of the users of JSON implementations that I
> know, is self-evidently incorrect.  -T

Assuming they exist, they're probably not here, making us the wrong
crowd, in the wrong forum.

BTW, I have done a few searches for how best to handle binary strings
in JS, and... I've found just one place so far indicating that binary
data in JSON strings works: comment #1 on stackoverflow question
17227998 (http://stackoverflow.com/questions/17227998/embedding-binary-data-in-web-page).
 This says nothing about how prevalent this is.

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Jul 2013 22:02:03 -0500

Oh, I just found this:

http://snia.org/sites/default/files/Multi-part%20MIME%20Extension%20v1.0g.pdf

The following is a quote from that doc, but I'm nto sure that it's
really indicative of anything, except awareness that JSON strings
"work" for binary data, just very inefficiently:

Multi-part MIME Transfer Extension

Overview

CDMI provides three methods by which the value of a data object may be
transferred between
CDMI clients and servers:
• UTF-8 encoding in JSON using a CDMI content type request/response,
• Base64 encoding in JSON using a CDMI content type request/response, and
• raw binary using a non-CDMI content type request/response.

UTF-8 encoding is sufficient for most text use cases, and using raw
binary transfer provided by
the non-CDMI PUT and GET operations is sufficient for some binary use
cases. However, there
is a need to be able to efficiently transfer binary data alongside
CDMI object metadata without
incurring the overhead of the UTF-8 or Base64 encoding and validation
required to represent
binary data in JSON.

This proposed extension adds the ability to use a multi-part MIME body
with CDMI to allow the
value to be included as raw binary data in a separate MIME part of a
single CDMI content type
request/response that does not require any encoding or validation of the data.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 18 Jul 2013 13:27:44 +0900

On 2013/07/18 6:34, Tim Bray wrote:
> we meet one, please?  I guess they’d have to be building  Browser-JS<=>

I'd of course agree with this sentiment.

> from them what their needs are, and what we can't put in 4267bis without

There's one possibility that just occurred to me, which is actually more 
in the future than in the past:

There is work going on for ECMAScript to get a 'binary' datatype. This 
will increase the amount of binary data being handled, and this will 
increase the pressure on JSON to carry binary data.

So it may be that even if there isn't much binary data in JSON out there 
at the moment, it may increase in the future.

We might argue that if we finish our work quickly, and make clear that's 
a very bad idea, we can avoid that (and that would be very good), but in 
a worst case scenario, that's not what might happen.

An alternative would be to think about a new version of JSON (JSON+?) 
that explicitly handles binary data, too.

Regards,   Martin.

---

From: "Markus Lanthaler" <[email-masked]>
Date: Thu, 18 Jul 2013 09:17:21 +0200

On Thursday, July 18, 2013 3:55 AM, Tim Bray wrote:
> I read the 18 messages in the thread and the discussion was at the
> level of theory. The constituency I’d like to hear from - actual
> developers who actually exchange actual JSON strings that can’t be
> expressed as sequences of Unicode codepoints - remains un-heard from.
> Remember that rough consensus & running code thing?  I’d really,
> really like to hear about instances of running code that depend on
> behavior that flatly contradicts the assertion in the RFC introduction
> that JSON strings are sequences of Unicode characters.
> 
> Yes, I understand that:
> - the ABNF allows arbitrary 16-bit quantities in strings, despite what
>  the intro says
> - Douglas Crockford says receiving software should just shut up and
> deal with naked surrogates
> 
> I just want to meet and understand the constituency we’re trying to
> serve by allowing behavior that, to 100% of the users of JSON
> implementations that I know, is self-evidently incorrect.  -T

+1, me too

--
Markus Lanthaler
@markuslanthaler

---

From: John Cowan <[email-masked]>
Date: Thu, 18 Jul 2013 11:30:11 -0400

> <https://mail.mozilla.org/pipermail/es5-discuss/2009-October/003385.html>

That message is misleading without its context.  It replies to a message
of mine which is about CouchDB, as I point out in my reply.  CouchDB
has nothing to do with ECMAscript.

What is more, in that thread I affirm that escaped unpaired surrogates
are permitted in JSON; what I am denying is that unescaped unpaired
surrogates are permitted in JSON-on-the-wire, which must be encoded in
Unicode (that is, in some Unicode character encoding scheme) per the RFC.

> ever will in a market with 5+ credible implementations.

+1

> correctness, but to document the status quo.

The trouble is that there are too many status quos.

-- 
John Cowan
        [email-masked]
                I am a member of a civilization. --David Brin

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 18 Jul 2013 15:52:47 +0000

>generator-parser combinations.

I would be perfectly fine with telling people that this use of JSON is
illegitimate, but still having it work sometimes if you can control both
endpoints - knowing that what you're doing is unlikely to widely
interoperate.

-- 
Joe Hildebrand

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 18 Jul 2013 19:37:43 +0200

>the assertion in the RFC introduction that JSON strings are sequences of

I am such a developer and you are failing to appreciate that making sure
code does not end up generating JSON with isolated surrogates is a lot
of hard work that the common libraries do not help you with. The onus is
rather on you to show us that running code actively sanatises strings to
keep isolated surrogates away from the JSON serialisers, or that typical
libraries do that for you, or refuse to serialise strings with them.

In ecmascript JSON.stringify does not mind isolated surrogates and there
are no sanitisation options. There are three options to do that yourself
by traversing an object replacing its strings as necessary, or creating
a copy to the same effect, replacing the surrogate escapes in the result
string, or using the replacer function feature. The code for the latter
in http://www.ietf.org/mail-archive/web/json/current/msg01170.html was

  JSON.stringify(obj, function(key, value) {
    if (typeof value == 'string') {
      return value.replace(/.../g, "\uFFFD");
    }
    return value;
  })

That is not something that people actually do in practise. People also
do not usually try to transmit isolated surrogate code points on purpose
but they do things like truncating strings to the first 140 16-bit code
units because that is how their String.substring function works, without
them ever having heard about Unicode surrogates.

In Perl the JSON modules JSON::XS and JSON::PP have no problem printing
out isolated surrogate escapes and there is no sanitisation option I've
been able to find. Am I supposed to regex them out of the serialized
JSON text? Can you write down the regex substitution code in a language
of your choice in under one minute and have it work correctly on first
try?

In Microsoft's .NET framework for the DataContractJsonSerializer I do
not see any documentation how to make it sanatise isolated surrogates,
and by default it happily passes them through as `\u....` escapes; and
in C# "\U0010FFFF".Length is 2 of course, so all substring operations
risk making JSON protocol breaking mistakes, if you had it your way.

Perhaps you can talk to the developers of Google's Gson library to tell
them about self-evidently incorrect behavior and an egregious bug you've
discovered in their code? What makes you think these actual developers
who generate isolated surrogates in JSON when somebody puts a non-BMP
character in a place they did not anticipate aren't actually almost all
of us?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Thu, 18 Jul 2013 11:06:36 -0700

OK, thanks Bjoern, we have the first actual real example of how naked
surrogates could reasonably arise in a real application:

On Thu, Jul 18, 2013 at 10:37 AM, Bjoern Hoehrmann <[email-masked]>wrote:

> I am such a developer

Actually, you haven’t quite convinced me that your software’s working
correctly depends on being able to stuff naked surrogates into JSON
strings.  But...

>   People also

And there’s the use case.  Nobody actually *wants* to interchange naked
surrogates, it’s just that there’s a cost to implementing things like
Substring() to avoid this happening, and it hasn’t been done.  BTW, I have
previously (10 years ago!) complained publicly about the broken behavior of
Java string-handling:
http://www.tbray.org/ongoing/When/200x/2003/04/30/JavaStrings#p-1

A genuinely new contribution to the conversation; thanks once again.  I’ll
have to think some to figure out whether this changes my opinion. -T

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 18 Jul 2013 19:33:35 +0000

On 7/18/13 12:06 PM, "Tim Bray" <[email-masked]> wrote:

>A genuinely new contribution to the conversation; thanks once again.
>I¹ll have to think some to figure out whether this changes my opinion. -T

Agree.  One possible approach would be to say that we use this
illegitimatization of isolated surrogates to explore how to make the
problem better over time.

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Wed, 24 Jul 2013 10:49:10 -0700

On Thu, Jul 18, 2013 at 11:06 AM, Tim Bray <[email-masked]> wrote:

> Substring() to avoid this happening, and it hasn’t been done.
>

So, someone jogged me and said “So have you anti-naked-surrogate types
changed your minds?”

After reflection, I ended up back where I started.

We still haven’t heard from anyone whose success depends on being able ot
use JSON strings for non-character uses involving naked surrogates.  We
have heard one scenario how a buggy-but-popular Java library could
accidentally create one.

I think that for 4267bis it remains an error condition that should be
warned against, and should we proceed to do i-json or equivalent, it should
become a MUST NOT; when you’re doing network interop, the implementer
burden of doing a post-truncation check when your library is known to be
buggy is not excessive.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 25 Jul 2013 07:36:50 +0000

On 7/24/13 11:49 AM, "Tim Bray" <[email-masked]> wrote:

>We still haven¹t heard from anyone whose success depends on being able ot
>warned against, and should we proceed to do i-json or equivalent, it
>should become a MUST NOT; when you¹re doing network interop, the
>implementer burden of doing a post-truncation check when your library is
>known to be buggy is not excessive.

Agree, with the further stipulation that I'd like to recommend some ways
to process the error, to make it clear why senders can't depend on a
receiving implementation choosing a specific approach.

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Thu, 25 Jul 2013 00:42:45 -0700

Good idea. Do you have a recommendation, Joe? Now would be a good time to
make it. -T

---

From: R S <[email-masked]>
Date: Thu, 25 Jul 2013 01:05:04 -0700

On Jul 24, 2013 10:50 AM, "Tim Bray" <[email-masked]> wrote:
>
> I think that for 4267bis it remains an error condition

That rhetoric is not convincing to me. People stick random bytes, the
results of hash functions, etc into JavaScript strings. Newer APIs do use
typed arrays and avoid this, but JSON doesn't have a direct analog.

More importantly, claiming that this is some kind of an error is
inaccurate. I don't think any of these seemingly credible implementors will
show up, because they don't have to. This working group can write down what
actually works or it can lie. The running code will still run, and the
rough consensus will remain.

It's common for IETF participants to think "rough consensus" refers to the
working group. Sometimes it does. For standards that are actually widely
used, the group is the wider Internet.

- Rob

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 25 Jul 2013 18:45:55 +0000

>Good idea. Do you have a recommendation, Joe? Now would be a good time to

Yes.  This is a start:

"""
2.5 Strings

The representation of strings is similar to conventions used in the C
   family of programming languages.  A string begins and ends with
   quotation marks.  All Unicode scalar values may be placed within the
   quotation marks except for the characters that must be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F).  Note that Unicode code points that are reserved for
UTF-16 surrogates (U+D800 through U+DFFF) are not valid Unicode scalar
values.

Any Unicode scalar value may be escaped.  If the Unicode scalar value
is in the Basic Multilingual Plane (U+0000 through U+D7FF and U+E000
through U+FFFF), then it may be represented as a six-character
sequence: a reverse solidus, followed by the lowercase letter u,
followed by four hexadecimal digits that encode the Unicode scalar
value's code point.  The hexadecimal letters A though
   F can be upper- or lowercase.  So, for example, a string containing
   only a single reverse solidus character may be represented as
   "\u005C".

Alternatively, there are two-character sequence escape
   representations of some popular characters.  So, for example, a
   string containing only a single reverse solidus character may be
   represented more compactly as "\\".

   To escape an Unicode scalar value that is not in the Basic Multilingual
   Plane, the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair that corresponds to the code point
for the Unicode scalar value.  So, for example, a string
   containing only the G clef character (U+1D11E) may be represented as
   "\uD834\uDD1E".

To compare strings for equality (for example, the uniqueness of names
in an object), escape sequences of any of the previous three kinds MUST
be normalized, then the strings are compared code point by code point.

If errors occur in string parsing, such as:
 - unescaped surrogates after decoding
 - unpaired escaped surrogate
 - invalid second character in a two-character escape sequence

implementations MAY make one of several different choices:
 - return an error and stop processing.  This is RECOMMENDED for
   security-sensitve deployments.
 - replace the offending code point with U+FFFD (REPLACEMENT CHARACTER).
   Strings that are intended to be human-readable might take this
   approach.
 - keep as much data as possible.  For example, some programming
   environments allow unpaired surrogates in their string
   representations, so unpaired surrogates may be retained.
   This approach is used by several existing implementations.

Because there are multiple valid choices for how to process invalid
data, senders SHOULD ensure that their strings are valid according to
the above rules.  Existing implementations are strongly encouraged to
be updated to always generate valid strings.  Examples of use cases to
test include:

 - string slicing operations that operate on 16-bit code units, rather
   than code points, which might generate an unpaired high surrogate
   at the beginning or end of the substring
 - unpaired surrogates that are valid in a given programming language

New implementations MUST NOT rely on a specific error implementation
choice for interoperability.

"""

-- 
Joe Hildebrand

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 26 Jul 2013 16:51:05 +0900

Some detailed comments inline:

On 2013/07/26 3:45, Joe Hildebrand (jhildebr) wrote:
> On 7/25/13 1:42 AM, "Tim Bray"<[email-masked]>  wrote:
>

Better add a pointer here to the list of these escapes.

> be normalized,

I don't think normalizing escape sequences is the right term. I'd say 
unescaping.

> then the strings are compared code point by code point.
>
>

The "after decoding" doesn't add anything here (except potential confusion).

>   - invalid second character in a two-character escape sequence

The first item is plural, the rest singular. I'd choose one or the other.

>     Strings that are intended to be human-readable might take this

Strings don't take any approach. -> "This approach might be taken for 
Strings intended to be human-readable." or "This approach is suitable 
for strings intended for display to humans." or some such.

>     This approach is used by several existing implementations.

I'd change "allow" to "tolerate".

>   - unpaired surrogates that are valid in a given programming language

I'd change the "valid" to something like "tolerated" or so.

Regards,   Martin.

---

From: Stefan Drees <[email-masked]>
Date: Fri, 26 Jul 2013 15:18:53 +0200

> Better add a pointer here to the list of these escapes.

esp. as the above notion of ' single reverse solidus ... represented 
more compactly as "\\" ' might irritate the reader, because the 
representation it compares with is **not** a string containing "\" but 
"\u005C" (as noted in the preceding paragraph).

> ...

{"Stefan": true}

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 26 Jul 2013 13:49:30 +0000

>Better add a pointer here to the list of these escapes.

Agree.

>I don't think normalizing escape sequences is the right term. I'd say

I wanted to allow for applications doing something that was more optimal
for their environment, but I really don't care that much.

>The "after decoding" doesn't add anything here (except potential

I'm fine with removing it.  The intent here was that of course surrogates
are ok before decoding if your wire format was UTF16.  Once we get to the
land of codepoints however, they can't exist.  CESU-8 isn't a valid
encoding for JSON.

>The first item is plural, the rest singular. I'd choose one or the other.

Nod.

>for strings intended for display to humans." or some such.

Agree.

>I'd change "allow" to "tolerate".

I like that a lot better.

>I'd change the "valid" to something like "tolerated" or so.

Agree.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Sat, 27 Jul 2013 08:49:19 +0000

>Some detailed comments inline:

OK, I think I got all of your comments included:

"""
2.5 Strings

Alternatively, there are two-character sequence escape
   representations of some popular characters, listed below:

 - \" U+0022 (QUOTATION MARK)
 - \\ U+005C (REVERSE SOLIDUS)
 - \/ U+002F (SOLIDUS)
 - \b U+0008 (BACKSPACE)
 - \f U+000C (FORM FEED)
 - \n U+000A (LINE FEED)
 - \r U+000D (CARRIAGE RETURN)
 - \t U+0009 (CHARACTER TABULATION)

So, for example, a string containing only a single reverse solidus
character may be represented more compactly than "\u005c" as "\\".

To compare strings for equality (for example, the uniqueness of names
in an object), escape sequences of any of the previous three kinds MUST
be unescaped, then the strings are compared code point by code point.

 - unescaped surrogate

implementations MAY make one of several different choices:
 - return an error and stop processing.  This is RECOMMENDED for
   security-sensitve deployments.
 - replace the offending code point with U+FFFD (REPLACEMENT CHARACTER).
   This approach might be taken for Strings intended for display to
   humans.
 - keep as much data as possible.  For example, some programming
   environments tolerate unpaired surrogates in their string
   representations, so unpaired surrogates may be retained.
   This approach is used by several existing implementations.

 - string slicing operations that operate on 16-bit code units, rather
   than code points, which might generate an unpaired high surrogate
   at the beginning or end of the substring
 - unpaired surrogates that are tolerated in a given programming language

"""

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Sun, 28 Jul 2013 00:25:51 -0700

This is really editorial, but could we please say “backslash” like normal
human beings instead of “reverse solidus”?  I have no idea what a solidus
is.  -T

---

From: John Cowan <[email-masked]>
Date: Sun, 28 Jul 2013 03:34:55 -0400

> human beings instead of “reverse solidus”?  I have no idea what a solidus

It's a Roman coin, later used as the Latin equivalent of "shilling".
Before decimalization the English wrote "two shillings sixpence" as
2/6, where the / was a stylized form of the old long s, for solidi
or shillings.  The names "solidus" and "reverse solidus" are ISO-speak
that Unicode adopted when it merged with ISO 10646.

-- 
We pledge allegiance to the penguin             John Cowan
and to the intellectual property regime         [email-masked]
for which he stands, one world under            http://www.ccil.org/~cowan
Linux, with free music and open source
software for all.               --Julian Dibbell on Brazil, edited

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Sun, 28 Jul 2013 08:01:44 +0000

On 7/28/13 9:34 AM, "John Cowan" <[email-masked]> wrote:

>Tim Bray scripsit:
>
>> This is really editorial, but could we please say ³backslash² like
>>normal
>> human beings instead of ³reverse solidus²?  I have no idea what a
>>solidus
>> is.  -T
>

I do not care much about how we refer to particular characters.  I just
copied the  canonical name from Unicode.  One approach would be to use the
correct name in the text, and the informal name in the comments to the
ABNF.

-- 
Joe Hildebrand