Subject: [Json] Scope: Wire format or runtime format?

From: Norbert Lindenberg <[email-masked]>
Date: Thu, 13 Jun 2013 15:47:38 -0700

In looking over older messages on this list, I found a message that made clear to me why we're having this endless discussion about Unicode surrogates - it's because we're not clear whether we're designing a wire format or a format that also for use at runtime:
http://www.ietf.org/mail-archive/web/json/current/msg00355.html

Some people are coming from the runtime point of view, especially ECMAScript, where it's accepted practice to use ill-formed UTF-16 or even non-text in strings. At least the ill-formed UTF-16 is legitimized by section 2.7 of the Unicode standard.

Other people are coming from the wire protocol point of view, where clean formats are expected, in particular well-formed Unicode code unit sequences according to section 3.9 of the Unicode standard.

So which one shall it be?

If we adopt the wire protocol point of view and require well-formed code unit sequences, then ECMAScript will have to define its own extension of JSON (which it has already by allowing JSON values at the top level).

If we adopt the runtime point of view and allow all code points as in RFC 4627, then there probably should be separate verbiage defining a restricted version for use over the wire.

Norbert

---

From: Nico Williams <[email-masked]>
Date: Thu, 13 Jun 2013 21:20:00 -0500

Well, JavaScript itself carries data and programs (but I repeat
myself) and gets carried around in network protocols all the time.  A
stretch, maybe, but to say that JavaScript is the language of the web
is not all that hyperbolic.

We might well end up with *two* JSONs.  I'd rather as much as possible
end up with just one.  The spec + best practices concept works for me
so far.

I don't really mind the compromises that actual deployments have
forced on JSON, be it re: duplicate names in objects or non-Unicode
content in what look like Unicode strings.  I was as shocked as others
for a bit, but it's passed :)

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 15 Jun 2013 11:27:30 -0400

> So which one shall it be?

Why not both? RFC 4627 deals with both; why should the update change to restrict that?

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Sat, 15 Jun 2013 08:56:15 -0700

I think the IETF, at least at the Apps area, is in the business of network
protocols, so I see our job here primarily as getting the wire-format
aspects right.

 -T

On Sat, Jun 15, 2013 at 8:27 AM, Paul Hoffman <[email-masked]> wrote:

> > In looking over older messages on this list, I found a message that made
> clear to me why we're having this endless discussion about Unicode
> surrogates - it's because we're not clear whether we're designing a wire
> format or a format that also for use at runtime:
> > Some people are coming from the runtime point of view, especially
> ECMAScript, where it's accepted practice to use ill-formed UTF-16 or even
> non-text in strings. At least the ill-formed UTF-16 is legitimized by
> section 2.7 of the Unicode standard.
> >
> > Other people are coming from the wire protocol point of view, where
> clean formats are expected, in particular well-formed Unicode code unit
> sequences according to section 3.9 of the Unicode standard.
> >
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: John Cowan <[email-masked]>
Date: Sat, 15 Jun 2013 12:06:48 -0400

> protocols, so I see our job here primarily as getting the wire-format

+1, but that still has at least two aspects, the character-level syntax
and the mapping to an entity body.

-- 
John Cowan  [email-masked]    http://ccil.org/~cowan
Big as a house, much bigger than a house, it looked to [Sam], a grey-clad
moving hill.  Fear and wonder, maybe, enlarged him in the hobbit's eyes,
but the Mumak of Harad was indeed a beast of vast bulk, and the like of him
does not walk now in Middle-earth; his kin that live still in latter days are
but memories of his girth and his majesty.  --"Of Herbs and Stewed Rabbit"

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Sat, 15 Jun 2013 20:35:32 +0000

>and the mapping to an entity body.

Is that only because of the multiple encodings?  For example, if JSON was
UTF8-only, could we define a purely concrete syntax of octet sequences?

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Sat, 15 Jun 2013 18:39:12 -0400

> UTF8-only, could we define a purely concrete syntax of octet sequences?

In principle, but that would make it difficult to embed JSON in HTML
or XML or JavaScript or another format defined by characters.

-- 
Business before pleasure, if not too bloomering long before.
        --Nicholas van Rijn
                John Cowan <[email-masked]>
                    http://www.ccil.org/~cowan

---

From: Tony Hansen <[email-masked]>
Date: Sat, 15 Jun 2013 19:33:20 -0400

> Why not both? RFC 4627 deals with both; why should the update change to restrict that?

I'm with Paul -- it needs to continue to do both.

IMO, there's 1) the character-based definition of the JSON format, the
one defined by the ABNF, and 2) an expression of that for over-the-wire
use as an application/json entity. The bulk of this document should be
focused on #1, and I think #2 can then be a fairly small section.

    Tony Hansen

---

From: Stefan Drees <[email-masked]>
Date: Sun, 16 Jun 2013 12:43:17 +0200

> focused on #1, and I think #2 can then be a fairly small section.

I second that POV. Until we started suggesting new titles for our 
well-beloved RFC 4627, we all focused mostly on #1 and took #2 for 
granted as necessary for hand-over "conventions".

Please keep these two aspects together in one document, and ... just let 
us **not** shoot too far over the approx. 2000 words RFC 4627 needed.
Most discussions - until now - convinced me that we will not be able to 
really add that much to it, so many additional words will be considered 
unreasonable and diluting.

An additional document even worse, seems like complete overhead to me.

Stefan Drees.

---

From: "Markus Lanthaler" <[email-masked]>
Date: Sun, 16 Jun 2013 12:59:39 +0200

> An additional document even worse, seems like complete overhead to me.

+1

--
Markus Lanthaler
@markuslanthaler

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 17 Jun 2013 10:59:21 +0900

On 2013/06/16 19:59, Markus Lanthaler wrote:
>> I second that POV.

+1.

But as far as I understand, these two were not the two that Norbert was 
talking about in the mail that started this thread. (More on that in a 
separate mail.)

>> Until we started suggesting new titles for our

+1, at least as long as we haven't tried seriously with one document.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 17 Jun 2013 13:17:46 +0900

Hello Norbert,

On 2013/06/14 7:47, Norbert Lindenberg wrote:
> In looking over older messages on this list, I found a message that made clear to me why we're having this endless discussion about Unicode surrogates -
 >it's because we're not clear whether we're designing a wire format or 
> http://www.ietf.org/mail-archive/web/json/current/msg00355.html

This is kind of the first time I have heard the term "format for use at 
runtime" in this context. Of course there are formats used at runtime 
(internal representations of number and strings, for example), but I 
think the term "runtime format" only confuses our discussion. In my 
understanding, JavaScript does not have or use JSON as a runtime format. 
If that has changed, then please tell us.

> Some people are coming from the runtime point of view, especially ECMAScript, where it's accepted practice to use ill-formed UTF-16 or even non-text in strings.
 >At least the ill-formed UTF-16 is legitimized by section 2.7 of the 
Unicode standard.

"Accepted practice" is probably going a bit too far, and giving the 
wrong impression. My understanding is that the Unicode standard accepts 
this for efficiency reasons, not because it's in anyway inherently 
useful. For ECMAScript, we have to add history to efficiency, but still 
I hope it's considered bad practice to actually use ill-formed UTF-16.

> Other people are coming from the wire protocol point of view, where clean formats are expected,
 > in particular well-formed Unicode code unit sequences according to 
section 3.9 of the Unicode standard.

 > If we adopt the wire protocol point of view

To me, it's very clear that we are describing (we are not designing; 
that has happened a long time ago, and very implicitly) a wire format, 
or something close to a wire format (somebody mentioned JSON embedded in 
an (e.g.) EUC-JP HTML page (*)).

> and require well-formed code unit sequences,

For practical reasons, I think we shouldn't go there. But we should make 
it very clear (in the base spec, not relegated to some "best practices" 
document) that using lone surrogates is a bad idea, and that senders and 
receivers MAY reject such data (e.g. because of security concerns).

> then ECMAScript will have to define its own extension of JSON

I very much hope we can avoid that. I very much hope that ECMAScript can 
tolerate that lone surrogates are often if not always a bad idea, even 
if they may sometimes happen for historical and efficiency reasons.

> (which it has already by allowing JSON values at the top level).

At least on the IETF side, as far as I know, this is still under discussion.

> If we adopt the runtime point of view and allow all code points as in RFC 4627, then there probably should be separate verbiage defining a restricted version for use over the wire.

I'm still at loss about what you mean by "runtime point of view". It 
seems very clear to me that neither the IETF version nor the ECMAScript 
version (in case they end up to differ) will in any way describe 
ECMAScript strings (or other datatypes) at runtime.

Regards,   Martin.

(*) It's not impossible for me to imagine something like JSON embedded 
in an (e.g.) EUC-JP HTML page, but I'm seriously wondering how 
widespread this is in practice. A simple case would be that a script 
says "take the content of this element and interpret it as JSON". But 
how widespread is that? Of course, we have a lot of stuff inside 
ECMAScript, but as soon as I'm in ECMAScript, something like
measurement = { 'unit': 'mm', 'amount': 15 }
is no longer JSON, it's just ECMAScript literal notation.

---

From: Norbert Lindenberg <[email-masked]>
Date: Mon, 17 Jun 2013 01:21:02 -0700

Hi Martin,

On Jun 16, 2013, at 21:17 , Martin J. Dürst wrote:

> This is kind of the first time I have heard the term "format for use at runtime" in this context. Of course there are formats used at runtime (internal representations of number and strings, for example), but I think the term "runtime format" only confuses our discussion. In my understanding, JavaScript does not have or use JSON as a runtime format. If that has changed, then please tell us.

Maybe you have a better term for it: What I mean in the ECMAScript context is the format that the functions JSON.parse and JSON.stringify parse and produce [1]. The format is based on the JSON grammar, represented in ECMAScript string values, that is, sequences of UTF-16 code units. ECMAScript doesn't specify wire formats (it doesn't even have any sort of I/O system), and has no control over what applications do with the strings before passing them to parse() or after obtaining them from stringify(). In many cases applications will get them from or send them to some I/O system (such as XMLHttpRequest in the browser or the file system module in Node.js), but they might modify them in between, or do something completely different.

> "Accepted practice" is probably going a bit too far, and giving the wrong impression. My understanding is that the Unicode standard accepts this for efficiency reasons, not because it's in anyway inherently useful. For ECMAScript, we have to add history to efficiency, but still I hope it's considered bad practice to actually use ill-formed UTF-16.

"Grudgingly accepted practice"? Unfortunately, considering it bad practice to use ill-formed UTF-16 or to represent JPEGs as string values doesn't make it go away.

> To me, it's very clear that we are describing (we are not designing; that has happened a long time ago, and very implicitly) a wire format, or something close to a wire format (somebody mentioned JSON embedded in an (e.g.) EUC-JP HTML page (*)).
> 

> For practical reasons, I think we shouldn't go there. But we should make it very clear (in the base spec, not relegated to some "best practices" document) that using lone surrogates is a bad idea, and that senders and receivers MAY reject such data (e.g. because of security concerns).

John Cowan pointed to RFC 3629, which prohibits the use of surrogate code points in UTF-8 [2]. Do you think that even for a wire format we can ignore that prohibition?

> I very much hope we can avoid that. I very much hope that ECMAScript can tolerate that lone surrogates are often if not always a bad idea, even if they may sometimes happen for historical and efficiency reasons.

The ECMAScript specification generally doesn't discuss good or bad ideas; it just specifies the behavior of implementations. The specified behavior is that unpaired surrogates in strings are passed through by both parse() and stringify(), and parse() unescapes escape sequences for all surrogate values, paired or not.

Personally, I agree that unpaired surrogates are in most cases a bad idea.

[1] http://ecma-international.org/ecma-262/5.1/#sec-15.12
[2] http://www.ietf.org/mail-archive/web/json/current/msg00831.html

Regards,
Norbert

---

From: John Cowan <[email-masked]>
Date: Mon, 17 Jun 2013 07:54:54 -0400

Norbert Lindenberg scripsit:

> Maybe you have a better term for it: What I mean in the ECMAScript
> context is the format that the functions JSON.parse and JSON.stringify
> parse and produce [1]. 

This simply is the description of JSON at the level of characters.

-- 
Almost all theorems are true,                   John Cowan <[email-masked]>
but almost all proofs have bugs.                http://www.ccil.org/~cowan
        --Paul Pedersen

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 18 Jun 2013 19:23:56 +0900

> This simply is the description of JSON at the level of characters.

Expressed as UTF-16 because that's what ECMAScript uses.

Regards,   Martin.

---

From: Jorge <[email-masked]>
Date: Tue, 18 Jun 2013 16:10:28 +0200

> Expressed as UTF-16 because that's what ECMAScript uses.

Note what JSON.stringify() produces is a subset of JSON, a subset of what JSON.stringify() accepts:

```javascript

a= '"\\'+'u2603"'
""\u2603""

b= '"\u2603"'
""☃""

JSON.parse(a) === JSON.parse(b)
true

Because JSON.stringify("☃") won't ever give you an ""\u2603"", even though it's as valid JSON as ""☃"", and .parse()s to the same value.
-- 
( Jorge )();