Subject: [Json] The parts of JSON (a possible aid to discussion)

From: Jacob Davies <[email-masked]>
Date: Thu, 6 Jun 2013 13:50:05 -0700

RFC 4267 does not clearly separate the various parts of JSON, relying
on intuitive interpretation. That has worked out pretty well and I
don't suggest changing that, but in this discussion I've found it
useful to think in terms of distinct sub-parts. They are:

1) The JSON data model.
2) The textual language described by the ABNF.
3) The byte-sequence format denoted by the MIME media type "application/json".

1) The JSON data model.

There is a description of the abstract data model in the original document:

   JSON can represent four primitive types (strings, numbers, booleans,
   and null) and two structured types (objects and arrays).

   A string is a sequence of zero or more Unicode characters [UNICODE].

   An object is an unordered collection of zero or more name/value
   pairs, where a name is a string and a value is a string, number,
   boolean, null, object, or array.

   An array is an ordered sequence of zero or more values.

This is clear except:

a) It omits a detailed description of null, boolean, and number values.

b) The description of strings as sequences of specifically Unicode
characters is in my opinion misplaced in a section that otherwise is
speaking in abstract terms. It is enough to say that a string is an
indefinite-length sequence of (abstract) characters. [I see Douglas'
last proposal for this section omits the mention of Unicode.]

c) Implementations must be free to represent this abstract model as a
concrete data model in any suitable way. For instance, they may not be
able to represent Unicode strings, or non-integer numbers, but that is
no reason they cannot work with some JSON models.

The abstract model ought to be unconcerned with Unicode or any such
temporal specificity, and the only thing the specification need say
about the concrete models of implementation is to warn that they may
vary in what ranges and sizes and kinds of values they can represent,
including which characters they can faithfully represent.

Perhaps something like:

"JSON is a language for representing models consisting of four
primitive types (strings, numbers, booleans, and null) and two
structured types (objects and arrays).

An object is an unordered collection of zero or more name/value pairs,
where a name is a string and a value is a string, number, boolean,
null, object, or array.

An array is an ordered sequence of zero or more values.

A number has an integer part of indefinite length, an optional decimal
fractional part of indefinite length, and an optional integer exponent
that may be positive or negative.

A string is a sequence of zero or more characters.

A boolean is either true or false.

A null has no further structure."

2) The textual language described by the ABNF.

This is most of the original specification. It very nearly consists of
language defined in terms of an abstract but unspecified full alphabet
containing abstract characters. The language is mostly concerned with
the arrangement of those characters drawn from the basic JSON alphabet
(required to be a subset of the full alphabet). There are a number of
proscribed characters that are not required to be in the full
alphabet, but if they are, may not literally appear in JSON texts
(some control characters) and some that are also not required, not
proscribed, but are restricted in where they may appear (certain
whitespace characters). And then additionally, in JSON strings all
other characters from the full alphabet that are not specifically
restricted nor proscribed may be literally included.

The language describes how to translate the abstract model to & from a
textual representation in the abstract full alphabet, without
specifying that full alphabet beyond the required, proscribed, and
restricted characters.

This includes the description of and required interpretation of escape
sequences, and here it says that \u escapes should be interpreted as
representing Unicode characters (or parts of surrogate pairs). That's
the only place in this part where anything concretely related to
Unicode needs to be invoked and it should not dictate that all
concrete model representations or all JSON texts or all byte
serializations be Unicode, only that the Unicode tables be used to
look up the character for \u escapes.

3) The byte-sequence format denoted by the MIME media-type "application/json".

This is the only part where the gritty details of format detection,
encoding, byte-order marks and so on are important.

It is connected to the textual language by requiring the
interpretation of "byte sequences of type application/json" as "JSON
texts where the full alphabet in use is Unicode, and the encoding is
UTF-8 (unless the application can determine otherwise)". Applications
could still be free to detect UTF-16 and UTF-32 formats, BOMs, or
whatever else they feel they can determine; all that is required is
that they must accept UTF-8 JSON texts if they claim to accept
application/json.

The standard should not be as hypertechnical as this, but it could
omit certain unnecessarily specific phrasings (e.g. of Unicode strings
in the abstract model and that JSON texts (in the abstract sense vs.
"documents of type application/json") be written in Unicode).
Interoperability should not be hurt by such omissions since the
definition of the media-type still completely defines how to interpret
a given sequence of bytes described as "application/json".

-Jacob

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 21:31:05 +0000

>last proposal for this section omits the mention of Unicode.]

Unicode code points are suitably abstract.  I agree that "character" is
probably not abstract enough.

>look up the character for \u escapes.

I doubt you need to use the tables, except much later in the process when
turning the parsed code points into glyphs.

I agree that in the abstract model, the only place you care about Unicode
is when reasoning about strings, however.  In the abstract model, true is
an atom with a particular semantic, not the sequence of code points
corresponding to the characters t,r,u, and e.

The rest of your message, while probably technically correct, effectively
says we're defining an infoset for JSON.  I don't like where that got XML,
but the multiple potential wire encodings for JSON may necessitate the
extra level of abstraction, if we're going to be super-clear.

Or we may elect to remain slightly fuzzy, and a lot easier to read.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 18:02:49 -0400

> says we're defining an infoset for JSON.  I don't like where that got XML,

Hey, I did the best I could, given vague and ever-changing requirements.
The MicroXML data model is far superior -- and it's represented using JSON.
See <https://dvcs.w3.org/hg/microxml/raw-file/tip/spec/microxml.html>.
You can't go wrong emulating James Clark when it comes to spec-writing.

-- 
Evolutionary psychology is the theory           John Cowan
that men are nothing but horn-dogs,             http://www.ccil.org/~cowan
and that women only want them for their money.  [email-masked]
        --Susan McCarthy (adapted)

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 22:15:17 +0000

>Hey, I did the best I could, given vague and ever-changing requirements.

I abjectly apologize.  I should have actually read the title block on the
infoset doc while I had it open.  Please don't take my statement as being
personal in any way.  I understand why we got there, I'm just not sure we
want to go all that way on JSON if we can avoid it.

>You can't go wrong emulating James Clark when it comes to spec-writing.

Interesting, I hadn't seen that.  Just removing namespaces probably
reduced the complexity tremendously.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Thu, 6 Jun 2013 18:21:34 -0400

Joe Hildebrand (jhildebr) scripsit:

> I abjectly apologize.  

No worries.  "I am your brother Joseph."  (Actually you're Joseph,
but whatever.)

> reduced the complexity tremendously.

Oh yes, and DTDs, and many other things.

-- 
Using RELAX NG compact syntax to        John Cowan <[email-masked]>
develop schemas is one of the simple    http://www.ccil.org/~cowan
pleasures in life....
        --Jeni Tennison                 <[email-masked]>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 07 Jun 2013 18:03:30 +0900

On 2013/06/07 6:31, Joe Hildebrand (jhildebr) wrote:
> On 6/6/13 2:50 PM, "Jacob Davies"<[email-masked]>  wrote:

 > including which characters they can faithfully represent.

The "which values they can represent" part may work (to some extent, but 
see below) for numbers, but not for strings. It doesn't matter *how* 
they are represented (somebody in China might want to use GB 18030), but 
it matters very much *what* is represented.

If you can show (or even just explain) an implementation of your JSON 
model that does not/cannot represent Unicode characters, but that is 
interoperable, then please do so.

>> serializations be Unicode,

You don't need to use an Unicode encoding for concrete representation. 
You may not need an Unicode encoding for byte serialization. But you 
need a concrete representation, and a byte encoding, than will handle 
all strings of Unicode code points and no other strings.

Regards,   Martin.

>> only that the Unicode tables be used to

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 13:03:12 -0400

"Martin J. Dürst" scripsit:

> You don't need to use an Unicode encoding for concrete
> representation. You may not need an Unicode encoding for byte
> serialization. But you need a concrete representation, and a byte
> encoding, than will handle all strings of Unicode code points and no
> other strings.

Not really.  The whole point of the escapes is so that you can encode
JSON in pure ASCII or any other encoding.  As long as the letters a,
b, c, d, e, f, l, n, r, s, t, and u, the digits 0-9, and the symbols
and punctuation marks left and right square bracket, left and right
brace, comma, period, minus sign, colon, double quote, and backslash
are representable, JSON is representable.  (You could design a 5-bit
encoding to do so.)

-- 
John Cowan    http://ccil.org/~cowan    [email-masked]
The work of Henry James has always seemed divisible by a simple dynastic
arrangement into three reigns: James I, James II, and the Old Pretender.
                --Philip Guedalla

---

From: Jacob Davies <[email-masked]>
Date: Fri, 7 Jun 2013 10:23:20 -0700

> Or we may elect to remain slightly fuzzy, and a lot easier to read.

I'm cool with fuzzy, and I think formalizing this model for public
consumption is probably pointless since it is intuitively quite clear
in the original RFC. But I thought it would be useful for the purposes
of this discussion to give some terms of reference for the different
parts of JSON-the-thing, since they are not broken out so distinctly
in the RFC.

I should have made the point regarding Unicode separately - speaking
of separating things - but that was that it is unnecessary to mention
Unicode anywhere except when discussing the interpretation of \u
escapes and in the discussion of the application/json media-type, and
that doing so in other places is a little confusing. (If I write a
JSON text on a whiteboard, am I using "Unicode characters"? This seems
like a map/territory issue - Unicode is a way of representing
characters as integers, but characters are the things under discussion
in the abstract model and textual language, not a particular mapping
of integers to characters.)

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 13:41:54 -0400

> of integers to characters.)

+1

Note that Unicode actually provides two things:  it maps every character
to an integer v, called a Unicode scalar value, such that 0 <= v < 56320
or 56575 < v <= 1114111; and it provides a variety of mappings between
Unicode scalar values and sequences of 8-bit, 16-bit, or 32-bit unsigned
integers, called code units.  This is the law; all else is commentary.

-- 
John Cowan   [email-masked]    http://ccil.org/~cowan
[R]eversing the apostolic precept to be all things to all men, I usually [before
Darwin] defended the tenability of the received doctrines, when I had to do
with the [evolution]ists; and stood up for the possibility of [evolution] among
the orthodox --thereby, no doubt, increasing an already current, but quite
undeserved, reputation for needless combativeness.  --T. H. Huxley