Subject: [Json] JSON mailing list and BoF

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 18 Feb 2013 20:01:07 +0000

We're planning on doing a BoF in Orlando to discuss starting up a JSON
working group.  The BoF is currently planned for Monday afternoon at 1300
in Carribean 6.  A very preliminary version of a charter can be found here:

http://trac.tools.ietf.org/wg/appsawg/trac/wiki/JSON

But obviously we'll need to build consensus on what it should actually
contain.  Please discuss on the [email-masked] mailing list:

https://www.ietf.org/mailman/listinfo/json

As a start, we know there are a couple of things we'd like to fix about
RFC 4627:

- move to standards track
- an ABNF indentation typo in section 2.2
- change SHOULD to MUST for the uniqueness of names in an object

A *very* important goal would be to minimize change, and to ensure strict
backward-compatibility.

There are likely adjunct JSON topics that would make sense to tackle once
we've got the experts together, but the goal would be to keep that list
short, focused, and general-purpose.

-- 
Joe Hildebrand

---

From: Francis Galiegue <[email-masked]>
Date: Mon, 18 Feb 2013 22:02:21 +0100

Hello,

> http://trac.tools.ietf.org/wg/appsawg/trac/wiki/JSON

JSON Schema is missing from this page (it has been updated recently
and is now three specifications).

-- 
Francis Galiegue, [email-masked]
Try out your JSON Schemas: http://json-schema-validator.herokuapp.com

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 18 Feb 2013 22:35:08 +0100

On Feb 18, 2013, at 21:01, "Joe Hildebrand (jhildebr)" <[email-masked]> wrote:

> adjunct JSON topics

One such topic might be roundtripping to/from some simple binary format for JSON-structured values.

draft-bormann-apparea-bpack-00.txt

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 18 Feb 2013 23:47:13 +0000

>draft-bormann-apparea-bpack-00.txt

As an individual, I'm +1 on that.  I love msgpack, and don't mind the
addition of UTF8 as a separate type.  Was frsyuki involved in the draft,
or at least know that it happened?

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 18 Feb 2013 23:52:24 +0000

>and is now three specifications).

As an individual.

I'd say that a way to unambiguously specify further standards that use
JSON seems reasonable.  Whether it's json-schema or json-content-rules, or
something else seems like a fertile ground for discussion.

I like Cyrus's argument that all we need is the ABNF of the JSON world,
and that any attempt to get more complicated than that will likely lead us
down the XSD rathole.

-- 
Joe Hildebrand

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 01:13:31 +0100

> and that any attempt to get more complicated than that will likely lead us

Well, I'd like to hear the reasoning behind this argument ;)

I have really struggled to make this specification useful, and
witnessing the users of my library alone, it looks like the pure
validation part of JSON Schema already has quite a few users...

I also make the best efforts not to repeat XSD's mistakes, and right
now JSON Schema is young enough that it cannot have made such mistakes
(I _think_). But in order not to repeat such mistakes, advice from
external people is of course needed.

Which is why reviews are always welcome!

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 01:07:36 +0000

>external people is of course needed.

Oh! Sorry, I didn't mean to imply that I didn't like json-schema, just
that there are several different approaches that have been proposed, and
it makes sense to have a conversation about what our requirements are and
if we think we're meeting them with one of those proposals.

Charter-wise, setting the requirement to be "as simple as possible, but no
simpler" is probably something we can all agree on; agreeing on what
constitutes "simple" is likely to be more complex. :)

> Which is why reviews are always welcome!

I certainly need to review the latest versions before having an opinion.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 01:12:30 +0000

On 2/18/13 5:26 PM, "Bjoern Hoehrmann" <[email-masked]> wrote:

>* Joe Hildebrand (jhildebr) wrote:
>A Working Group will also likely have to look at revising the rules to
>detect the character encoding in application/json resources (there is a
>lack of consensus whether to auto-detect UTF-32 and whether to honour a
>Unicode signature, whether to support UTF-32 when it is detected, and so
>on, partly due to how people load JSON resources; a generic "load text"
>API for instance might detect and remove a Unicode signature before the
>data is passed to a JSON parser; this used to be true for XMLHttpRequest
>for instance, I haven't checked the current situation there though).

(individual)

Yeah, that probably needs to be dealt with.  If 4627 had been written as
"JSON SHALL always be encoded as UTF-8 when transmitted or stored", then
this would not be a problem.

I wonder if it's too late for that?  If we said that the
backward-compatibility rule was that any document that was valid 4627bis
would be correctly parsed by a 4627 parser, going all-in on UTF-8 would be
ok.  Since this is currently valid and proposed to no longer be valid:

{"foo": "bar", "foo": "baz"}

clearly the reverse is not going to be true anyway.

-- 
Joe Hildebrand

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 02:15:14 +0100

On Tue, Feb 19, 2013 at 2:07 AM, Joe Hildebrand (jhildebr)
<[email-masked]> wrote:
[...]
> if we think we're meeting them with one of those proposals.

I'd certainly like to have a digest of this discussion! I have my own
needs, and even my own implementation, so I cannot be partial here --
just be a witness on my use case.

> I certainly need to review the latest versions before having an opinion.

Looking forward to this!

Have fun,

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Feb 2013 02:41:17 +0100

>this would not be a problem.

A practical problem is that some people include a Unicode Signature in
their JSON documents, and their code works in some environments either
by accident or because someone made a deliberate choice to support the
Unicode Signature even though the RFC does not permit one, but it would
break in more strict implementations. I would expect the Working Group
to investigate whether the "UTF-8 BOM" should be allowed and how JSON
documents with one should be handled, even if it decides to limit JSON
to UTF-8.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 18 Feb 2013 18:44:36 -0700

On 2/18/13 6:41 PM, Bjoern Hoehrmann wrote:
>>> A Working Group will also likely have to look at revising the
>>> rules to detect the character encoding in application/json
>>> resources (there is a lack of consensus whether to auto-detect
>>> UTF-32 and whether to honour a Unicode signature, whether to
>>> support UTF-32 when it is detected, and so on, partly due to
>>> how people load JSON resources; a generic "load text" API for
>>> instance might detect and remove a Unicode signature before
>>> the data is passed to a JSON parser; this used to be true for
>>> XMLHttpRequest for instance, I haven't checked the current
>>> situation there though).
>> 
>> (individual)
>> 
>> Yeah, that probably needs to be dealt with.  If 4627 had been
>> written as "JSON SHALL always be encoded as UTF-8 when
>> transmitted or stored", then this would not be a problem.
> 
> A practical problem is that some people include a Unicode Signature
> in their JSON documents, and their code works in some environments
> either by accident or because someone made a deliberate choice to
> support the Unicode Signature even though the RFC does not permit
> one, but it would break in more strict implementations. I would
> expect the Working Group to investigate whether the "UTF-8 BOM"
> should be allowed and how JSON documents with one should be
> handled, even if it decides to limit JSON to UTF-8.

Yeah, limiting things to UTF-8 these days would be a good thing. But,
as Bjoern notes, we might want to have a section about
interoperability with existing implementations.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 18 Feb 2013 18:02:58 -0800

> clearly the reverse is not going to be true anyway.

If we are supposed to be keeping backwards compatibility, then yes, it's too late. The same is true for changing SHOULD not have duplicate names in objects.

OTOH, if we want a cleaner standard going forwards, by all means let's take out the over-cautious SHOULDs, and make a clear statement in the Introduction where we changed the requirements.

I'm strongly in favor of the latter.

--Paul Hoffman

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Feb 2013 14:12:08 +1100

> I'm strongly in favor of the latter.

you ARE going to get Crockford involved in this process somehow, right?

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Feb 2013 01:26:37 +0100

---

From: Tim Bray <[email-masked]>
Date: Mon, 18 Feb 2013 23:05:42 -0800

Easily solved: “In all instances when JSON is transmitted over a network,
it MUST be encoded in UTF-8; a charset argument on the Content-type MAY be
provided, but its value MUST be utf-8.”

 -T

On Mon, Feb 18, 2013 at 4:26 PM, Bjoern Hoehrmann <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 15:21:46 +0000

>you ARE going to get Crockford involved in this process somehow, right?

I contacted him before the list was formed, but I'll reach out again to
make sure he's got the right coordinates.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 15:25:08 +0000

On 2/19/13 12:05 AM, "Tim Bray" <[email-masked]> wrote:

>Easily solved: ³In all instances when JSON is transmitted over a network,
>it MUST be encoded in UTF-8; a charset argument on the Content-type MAY
>be provided, but its value MUST be utf-8.²

Could we change section 3 to explain that if you want to liberal in what
you accept you can parse according to the older rules (including dealing
with a BOM), but that you MUST always generate UTF-8 with no BOM if you're
going to be 4627bis compliant?

-- 
Joe Hildebrand

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Tue, 19 Feb 2013 10:59:27 -0500

> going to be 4627bis compliant?

I think that is reasonable, and it maintains the backwards compatibility you stressed as critical to this effort.

Gonzalo

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 19 Feb 2013 17:39:11 +0100

> or at least know that it happened?

I tried to involve him.
Frsyuki doesn't like what I did with the binary strings vs. UTF-8, though.

I didn't quite catch whether he is just focusing on maintaining compatibility with what's out there (today's msgpack) or whether he really doesn't see why that is a good idea.
(For me, maintaining 100 % compatibility won't work anyway, because in the end that won't be the only change we'll want to make.  
If we look a bit outside the space that msgpack is being applied to today, we might want to support, say, 16-bit floating point.)

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 17:31:42 +0000

On 2/19/13 9:39 AM, "Carsten Bormann" <[email-masked]> wrote:

>If we look a bit outside the space that msgpack is being applied to
>today, we might want to support, say, 16-bit floating point.)

and potentially Date.

If we were going to standardize this, the IETF would need full change
control.

-- 
Joe Hildebrand

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 18:37:22 +0100

On Tue, Feb 19, 2013 at 6:31 PM, Joe Hildebrand (jhildebr)
<[email-masked]> wrote:
[...]
> If we were going to standardize this, the IETF would need full change

Another unrelated point: I see no reason why a transmitted JSON
document (called a "JSON text") has to be a "container" and not any
JSON value.

What was the motivation behind this in the RFC in the first place?

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Tue, 19 Feb 2013 20:33:09 +0000

Another topic that might worth discussing is canonicalization.  It's come up in JOSE a couple of times[1], and it would be helpful to standardize an approach.

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

[1] The most recent discussing starts at < http://www.ietf.org/mail-archive/web/jose/current/msg01575.html >, instigated by < http://www.ietf.org/mail-archive/web/jose/current/msg01553.html >

On Feb 18, 2013, at 1:01 PM, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 20:47:09 +0000

On 2/19/13 1:33 PM, "Matt Miller (mamille2)" <[email-masked]> wrote:

>Another topic that might worth discussing is canonicalization.  It's come
>up in JOSE a couple of times[1], and it would be helpful to standardize
>an approach.

(individual)

Do you think draft-staykov-hu-json-canonical-form is a valid starting
point?  My biggest beef with it is that the number syntax uses capital E
and JavaScript's toExponential() generates a lowercase e.

-- 
Joe Hildebrand

---

From: Mike Jones <[email-masked]>
Date: Tue, 19 Feb 2013 20:47:02 +0000

I'm strongly against canonicalization.  The XML canonicalization experience was horrible and resulted in more interop bugs than any other aspect of XML DSIG, XML ENC, etc.  Let's not repeat the mistakes of our elders. ;-)

I also haven't seen a clear use case that canonicalization solves that can't be more easily solved another way.

				-- Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Matt Miller (mamille2)
Sent: Tuesday, February 19, 2013 12:33 PM
To: Joe Hildebrand (jhildebr)
Cc: [email-masked]; [email-masked]
Subject: Re: [apps-discuss] JSON mailing list and BoF

- m&m

On Feb 18, 2013, at 1:01 PM, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> working group.  The BoF is currently planned for Monday afternoon at 
> 1300 in Carribean 6.  A very preliminary version of a charter can be found here:
> 
> As a start, we know there are a couple of things we'd like to fix 
> about RFC 4627:
> 
> A *very* important goal would be to minimize change, and to ensure 
> strict backward-compatibility.
> 
> There are likely adjunct JSON topics that would make sense to tackle 
> once we've got the experts together, but the goal would be to keep 
> that list short, focused, and general-purpose.
> 
> --
> Joe Hildebrand
> 
> 
>

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 21:27:38 +0000

On 2/19/13 1:47 PM, "Mike Jones" <[email-masked]> wrote:

>I'm strongly against canonicalization.  The XML canonicalization
>experience was horrible and resulted in more interop bugs than any other
>aspect of XML DSIG, XML ENC, etc.  Let's not repeat the mistakes of our
>elders. ;-)
>

I somewhat agree, but have you at least read
draft-staykov-hu-json-canonical-form?  It's pretty straightforward, and
nowhere near as scary as xmlenc.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 21:28:59 +0000

>can't be more easily solved another way.

I needed it for a light use case recently; I wanted to use a serialized
object as the key into a hash.

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 13:34:50 -0800

re draft-staykov-hu-json-canonical-form:
Looks sensible. One gripe:  Why can’t you sign NaN and INF?  They do in
fact  occur in the field, and it’s not as though a noncanonical
representation is possible.

-T (who still hasn’t decided if this is actually a good idea)

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 15:41:27 -0600

> nowhere near as scary as xmlenc.

Indeed.

Lexicographic sorting... makes me wonder about Unicode normalization.
Before your heads explode, what is the ECMAScript rule regarding
normalization of object keys?  Are the two normalization forms of
'foó' allowed as distinct keys in an object?

Nico
--

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Tue, 19 Feb 2013 21:42:16 +0000

> and JavaScript's toExponential() generates a lowercase e.

Assuming I think canonicalization is something that ought to be done, I think it is a decent start that needs to be expanded.  The first thing that came to my mind is string normalization, and how that relates to escape sequences.

However, while not as violently opposed as some, I am hesitant.  XML canonicalization has bitten me hard in the past, so I am carrying a bias into this conversation.  Maybe a starting point can be "don't rely on canonicalization unless you really really really have to," and we can come up with a set of those "really really really have to" scenarios.

- m&m

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 13:53:07 -0800

I would argue that normalization should be out of scope.  I.e. the two
forms of “foó” are different strings, that’s all.
“Doctor! It hurts when I do this!”
“So, don’t do that!”

-T

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 21:55:37 +0000

On 2/19/13 2:34 PM, "Tim Bray" <[email-masked]> wrote:

>Looks sensible. One gripe:  Why can¹t you sign NaN and INF?  They do in
>fact  occur in the field, and it¹s not as though a noncanonical

I can figure out how to generate a -INF, but how do I get a -NaN?

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2013 13:59:41 -0800

The JSON list seems active already. Do we need to have this thread on both lists?

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 15:59:56 -0600

On Tue, Feb 19, 2013 at 3:53 PM, Tim Bray <[email-masked]> wrote:
> I would argue that normalization should be out of scope.  I.e. the two forms
> of “foó” are different strings, that’s all.

That's fair.  In that case this I-D looks simple enough that I can't
really oppose it.  I do think we should give advice as to how to avoid
the need for canonicalization (c14n?) in the first place.

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 23:00:59 +0100

> “Doctor! It hurts when I do this!”

(as a not very well educated individual on this matter...)

If _that_ is what normalization is about, then I strongly condemn it
being even a subject at all.

The current RFC does a pretty good job at defining what a JSON String
can be, fabricating some sort of injective function so that one JSON
value is equal to another is sick.

OK, I say that, but on the other hand, JSON Schema mandates that JSON
values "1.0" and "1" be considered equal for numeric validation. This
could also be viewed as a form of normalization...

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 16:01:09 -0600

> I can figure out how to generate a -INF, but how do I get a -NaN?

In JS?  1/0 -> INF, 0/0 -> NaN.

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 23:05:08 +0100

On Tue, Feb 19, 2013 at 10:55 PM, Joe Hildebrand (jhildebr)
<[email-masked]> wrote:
[...]
> I can figure out how to generate a -INF, but how do I get a -NaN?

The current JSON specification doesn't mandate any limit to the
precision and/or scale of numeric instances, and I was particularly
attentive to this aspect of JSON when writing the JSON Schema specs,
since my first use case for using JSON Schema was the ability to use
keywords such as "minimum", "maximum", "multipleOf" to validate
arbitrary numerical JSON instances. The JSON Schema validation specs
(OK, these are my words) says this:

http://tools.ietf.org/html/draft-fge-json-schema-validation-00#section-3.2

I _do believe that JSON itself should not define NaN. JSON can express
arbitrary data and should not care about the capabilities of language
A or B in a transacation. Only recommendations and/or warnings can be
given at that point.

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 16:07:26 -0600

> value is equal to another is sick.

Oh, well, the RFC (4627) says nothing about normalization.  And if you
think the notion is sick wait till you hear the details!  But as you
read what you find when you search for "Unicode normalization" you
should keep one thing in mind: this sickness isn't Unicode's fault --
it's our (humans', that is) fault.

Indeed.

Nico
--

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:08:05 +0000

[Thank you Paul; removing apps-discuss@]

> could also be viewed as a form of normalization...

And furthermore, "foó" versus "\u0066\u006f\u00f3" (or "\u0066\u006f\u006f\u00b4"?).  I think most of us clearly think the former is better than the latter, but it does need to be called out.

I also do think some level of discussion on unicode normalization, even if the result is "this is out-of-scope," needs to take place.

- m&m

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:18:14 +0000

>The JSON list seems active already. Do we need to have this thread on

Nope.  After this message, I'll start pruning and re-directing, since I
think we've got a quorum on the json list.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:19:46 +0000

On 2/19/13 3:01 PM, "Nico Williams" <[email-masked]> wrote:

>In JS?  

Yes.

>1/0 -> INF, 0/0 -> NaN.

-1/0 -> -Infinity
? -> -Nan?

-- 
Joe Hildebrand

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 23:20:10 +0100

On Tue, Feb 19, 2013 at 11:07 PM, Nico Williams <[email-masked]> wrote:
[...]
>> could also be viewed as a form of normalization...

Note however that JSON Schema is not the only I-D in that case. JSON
Patch says this too
(http://tools.ietf.org/html/draft-ietf-appsawg-json-patch-10#section-4.6).

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 14:20:20 -0800

Oops, I was wrong, NaN & INF aren’t in JSON.  D’oh.  -T

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:23:49 +0000

And reading your mail more carefully, I thought "sign" here meant +/-,
which is why that other part of this thread is down in the weeds.

On 2/19/13 3:20 PM, "Tim Bray" <[email-masked]> wrote:

>Oops, I was wrong, NaN & INF aren¹t in JSON.  D¹oh.  -T
>
>-T (who still hasn¹t decided if this is actually a good idea)
>
>
>
>On Tue, Feb 19, 2013 at 1:27 PM, Joe Hildebrand (jhildebr)

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 14:23:51 -0800

Options for \uxxxx
- forbid it
- argue that comparisons MUST be Unicode-codepoint by Unicode-codepoint,
and specify that \uxxxx specifies a single codepoint

Latter is probably better. -T

On Tue, Feb 19, 2013 at 2:08 PM, Matt Miller (mamille2)
<[email-masked]>wrote:

> forms
> And furthermore, "foó" versus "\u0066\u006f\u00f3" (or
> "\u0066\u006f\u006f\u00b4"?).  I think most of us clearly think the former
> is better than the latter, but it does need to be called out.
>
> I also do think some level of discussion on unicode normalization, even if
> the result is "this is out-of-scope," needs to take place.
>
>
> - m&m
>

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 23:27:34 +0100

On Tue, Feb 19, 2013 at 11:23 PM, Tim Bray <[email-masked]> wrote:
> Options for \uxxxx
> - forbid it
> - argue that comparisons MUST be Unicode-codepoint by Unicode-codepoint, and
> specify that \uxxxx specifies a single codepoint
>

imho, option 2 is what "should" happen, but isn't this what is
currently enforced already?

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:29:43 +0000

On 2/19/13 3:08 PM, "Matt Miller (mamille2)" <[email-masked]> wrote:

>"\u0066\u006f\u006f\u00b4"?).  I think most of us clearly think the
>former is better than the latter, but it does need to be called out.
>
>I also do think some level of discussion on unicode normalization, even
>if the result is "this is out-of-scope," needs to take place.

One of the things we should also consider adding to the scope of a
potential WG is a set of design guidelines for protocols that use JSON.
One of those guidelines might be to avoid using human-language text as the
keys in an object, for this and other reasons.

However, I immediately thought of a counter-example: if you wanted to
encode a translation dictionary, you might want to do this:

{
  "Hello": {
    "en-US": "Howdy"
  }
}

It might be awkward to do this instead:

[
  {
    "word": "Hello",
    "translations": {
      "en-US": "Howdy"
    }
  }
]

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:31:48 +0000

\u0000 is an exception.

On 2/19/13 3:23 PM, "Tim Bray" <[email-masked]> wrote:

><[email-masked]> wrote:
>
>>> of ³foó² are different strings, that¹s all.
>>> ³Doctor! It hurts when I do this!²
>>> ³So, don¹t do that!²
>>>
>>

-- 
Joe Hildebrand

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:32:18 +0000

> Latter is probably better. -T

Assuming that canonicalization is as much about what's on the wire as what is logically, I don't think the latter is enough.

However, the former might be acceptable with some *very* limited exceptions:

* NULL (\u0000)
* QUOTE (\u0022)
* REVERSE SOLIDUS (\u005c")

There's probably a couple of others that have to be there, but I think that list is rather short.

- m&m

---

From: Francis Galiegue <[email-masked]>
Date: Tue, 19 Feb 2013 23:39:01 +0100

> \u0000 is an exception.

Why? It is a perfectly legal Unicode code point...

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 14:40:41 -0800

JSON, unlike XML, does not exclude U+0000

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Tue, 19 Feb 2013 22:42:43 +0000

> Why? It is a perfectly legal Unicode code point...

He means "\u0000 is an exception to Tim's proposed 'forbid escaping' rule".

- m&m

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 00:02:15 +0100

> JSON, unlike XML, does not exclude U+0000

And is there any valid reason as to why JSON should forbid this
particular code point? IMHO, no. Although it will certainly be a
_very_ rare code point in real life, I see no valid reason to
explicitly disallow it since it _is_ representable, after all.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 23:08:32 +0000

>He means "\u0000 is an exception to Tim's proposed 'forbid escaping'

Exactly.  It's the only code point that I know of that MUST be escaped
using \uxxxx notation when serialized (either canonically or otherwise).
All of the other codepoints that need to be escaped have \x versions.

-- 
Joe Hildebrand

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 00:13:15 +0100

On Wed, Feb 20, 2013 at 12:08 AM, Joe Hildebrand (jhildebr)
<[email-masked]> wrote:
[...]
> All of the other codepoints that need to be escaped have \x versions.

OK, but why should it be considered illegal at all? Any ASCII
character is representable by such a sequence, it just happens that 0
can where ASCII can't... I don't see this as a reason to outlaw this
particular sequence... Or there is something I don't understand in
your argument.

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 15:42:46 -0800

I haven’t heard anyone saying U+0000 should be illegal in JSON.  It’s too
late to redesign JSON. -T

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Feb 2013 23:52:34 +0000

Nor was I suggesting we do so.

I was suggesting that if you want to get \u0000 - \u001f, excluding
\u0008(\b), \u0009(\t), \u000a(\n), \u000c(\f), and \u000d(\r) into the
canonicalized output, you have to use the \u notation, which means we
can't outlaw \u notation entirely in the canonicalized form, as some of us
thought Tim had suggested in the first place.

Note, I was wrong about \u0000 being the only codepoint we care about.
>From section 2.5 of 4627:

"All Unicode characters may be placed within the
   quotation marks except for the characters that must be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F)."

On 2/19/13 4:42 PM, "Tim Bray" <[email-masked]> wrote:

>I haven¹t heard anyone saying U+0000 should be illegal in JSON.  It¹s too
>On Tue, Feb 19, 2013 at 3:13 PM, Francis Galiegue
><[email-masked]> wrote:
>
>Try out your JSON Schemas:
>http://json-schema-validator.herokuapp.com
><http://json-schema-validator.herokuapp.com>
>
>
>
>
>
>
>
>

-- 
Joe Hildebrand

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 16:02:05 -0800

Yes, I guess I did sort of strawman-suggest banning \uxxxx.  Which clearly
doesn't fly.  So, two suggestions:

- \uxxxx is only allowed for chars that must be escaped per the JSON spec
- \uxxxx is freely allowed, but the \uxxxx is considered to represent a
single codepoint, and all comparison/hashing operations have to be
conducted codepoint-by-codepoint

I think I probably support the first, because it does allow comparison
using strcmp() or equivalent. That's assuming that c14n is in fact worth
doing. -T

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 20 Feb 2013 00:08:38 +0000

On 2/19/13 5:02 PM, "Tim Bray" <[email-masked]> wrote:

>Yes, I guess I did sort of strawman-suggest banning \uxxxx.  Which
>clearly doesn't fly.  So, two suggestions:
>
>

+1 to the first approach, and also +1 about not knowing whether we need to
do c14n or not yet.

-- 
Joe Hildebrand

---

From: "Paul C. Bryan" <[email-masked]>
Date: Tue, 19 Feb 2013 16:17:59 -0800

Should two representations of the same JSON (string) value be c14n'ed
differently? I think not*.

If not, then I think comparisons would need to occur by parsing JSON
strings into Unicode codepoints.

Paul

* Why not? Mostly because I foresee many operations such as signature
verification occurring on already-parsed JSON values, so how it was
represented when it was serialized should (ideally) be moot.

On Tue, 2013-02-19 at 16:02 -0800, Tim Bray wrote:
> - \uxxxx is freely allowed, but the \uxxxx is considered to represent
> a single codepoint, and all comparison/hashing operations have to be
> using strcmp() or equivalent. That's assuming that c14n is in fact
> worth doing. -T
> 
> 
> 
> 
> 
> 
> On Tue, Feb 19, 2013 at 3:52 PM, Joe Hildebrand (jhildebr)
>         >>>He means "\u0000 is an exception to Tim's proposed 'forbid
>         escaping'
>         >>>rule".
>         >>

---

From: Mike Jones <[email-masked]>
Date: Wed, 20 Feb 2013 00:23:44 +0000

As background, the JOSE specs require that equality comparisons for members be done as a comparison of Unicode code points, with no normalization or case folding performed.  See http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-08#section-8.3.

                                                                -- Mike

From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul C. Bryan
Sent: Tuesday, February 19, 2013 4:18 PM
To: [email-masked]
Subject: Re: [Json] [apps-discuss] JSON mailing list and BoF

Should two representations of the same JSON (string) value be c14n'ed differently? I think not*.

If not, then I think comparisons would need to occur by parsing JSON strings into Unicode codepoints.

Paul

* Why not? Mostly because I foresee many operations such as signature verification occurring on already-parsed JSON values, so how it was represented when it was serialized should (ideally) be moot.

On Tue, 2013-02-19 at 16:02 -0800, Tim Bray wrote:
Yes, I guess I did sort of strawman-suggest banning \uxxxx.  Which clearly doesn't fly.  So, two suggestions:

- \uxxxx is only allowed for chars that must be escaped per the JSON spec
- \uxxxx is freely allowed, but the \uxxxx is considered to represent a single codepoint, and all comparison/hashing operations have to be conducted codepoint-by-codepoint

I think I probably support the first, because it does allow comparison using strcmp() or equivalent. That's assuming that c14n is in fact worth doing. -T

On Tue, Feb 19, 2013 at 3:52 PM, Joe Hildebrand (jhildebr) <[email-masked]<mailto:[email-masked]>> wrote:
Nor was I suggesting we do so.

On 2/19/13 4:42 PM, "Tim Bray" <[email-masked]<mailto:[email-masked]>> wrote:

><[email-masked]<mailto:[email-masked]>> wrote:
>
><[email-masked]<mailto:[email-masked]>> wrote:
>[...]
>>>
>>>
>Francis Galiegue, [email-masked]<mailto:[email-masked]>

--
Joe Hildebrand

json mailing list

[email-masked]<mailto:[email-masked]>

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 01:29:51 +0100

> If not, then I think comparisons would need to occur by parsing JSON strings

+1 on that. I see no reason why "a" and "\u0061" should be treated
differently at all.

Or I have completely misundertood the point...

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 18:31:34 -0600

On Tue, Feb 19, 2013 at 6:23 PM, Mike Jones <[email-masked]> wrote:
> As background, the JOSE specs require that equality comparisons for members
> be done as a comparison of Unicode code points, with no normalization or
> case folding performed.  See
> http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-08#section-8.3.

Was that because normalization is hard, mostly useless (most input
strings will already be NFC anyways), or for some other reason?

Nico
--

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 20 Feb 2013 11:57:54 +1100

We should avoid the need to canonicalize JSON whenever possible, but there are enough efforts to define a canonical form that it would be worth standardizing one.

Strings:
Escaping is mandatory for controls chars, double quote, and backslash (%x00-1F / %x22 / %x5C).
The simplest string canonicalization rule would be to escape those 34 chars and no others. A rule that might be slightly nicer for people (and hence worth the few extra lines of code) would be to always use the 7 \x escapes for those 7 chars, always use \uxxxx for the rest of %x00-1F, and never use it for any other chars. I would be tempted to drop \/ from the list of seven (only in the canonicalization rules), because / is so common in URIs etc.

Objects:
Sorting string names to canonicalize an object needs a few more words.
Presumably sorting occurs on the logical strings, not the (canonically) encoded versions. So {"a\"b":1,"a#b":2} is in canonical form ("(U+22) < #(U+23) < \(U+5C)).
Presumably sorting uses Unicode code points, not UTF-16 words, or UTF-8 bytes. So {"\uFFE0\uFFE1":3,"\uD834\uDD1E":4} is in the right order (0xFFE0 < 0x01D11E), thought the canonical form would use UTF-8 not \uxxxx for the 3 characters.
[The 21-byte canonical form would be (in hex):
7B 22 EFBFA0 EFBFA1 22 3A 33 2C 22 F09D849E 22 3A 34 7D]

Numbers:
0, 1, 1e3, 2.334e-5, 1.5e6 are the sort of canonical form numbers should have. I don’t like 0.0E0 for zero as per draft-staykov-hu-json-canonical-form-00 -- a person would never write that. That draft also allows any number of trailing 0’s (eg 1.200000e-2), which is a bug. A canonical form should drop the exponent when it is zero, and drop the decimal point when there is nothing after it.
A regex for numbers in canonical form:

0|-?[1-9](\.[0-9]*[1-9])?(e-?[1-9][0-9]*)?

So draft-staykov-hu-json-canonical-form needs a few changes in my mind, but is as good a starting point as any.

--
James Manger

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2013 17:00:05 -0800

One only needs to canonicalize at the time when one is comparing two objects. We should say *nothing* that suggests that JSON sent on the wire should be canonicalized at all.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 19:11:51 -0600

> One only needs to canonicalize at the time when one is comparing two objects. We should say *nothing* that suggests that JSON sent on the wire should be canonicalized at all.

When to canonicalize:

 - when you're signing something that a peer has to reconstruct independently
 - when you want to compare with memcmp()

Note that that's not the only way to compare JSON data!  One could
parse into native objects in whatever programming language one is
using and then proceed to compare bit by bit.

Note too that equality comparison does require saying something about
Unicode normalization.  The simplest thing to say about that would be
that there are two types of comparison: naive, which uses
memcmp()/strcmp(), and normalization-insensitive (which internally
might normalize then compare or work as a normalization-insensitive
comparison; the latter performs better).

Note that the first use of c14n given above also requires saying
something about normalization...

When to not canonicalize: any other time.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2013 17:21:18 -0800

> - when you're signing something that a peer has to reconstruct independently

Canonicalizing is part of the signature process, not part of the JSON creation process. The spec that specifies how to sign JSON will pick what to do about canonicalization; the JOSE WG is dealing with this.

> - when you want to compare with memcmp()

Sure. That's what I said above.

--Paul Hoffman

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 02:28:14 +0100

On Wed, Feb 20, 2013 at 2:21 AM, Paul Hoffman <[email-masked]> wrote:
[snip]
> Canonicalizing is part of the signature process, not part of the JSON creation process. The spec that specifies how to sign JSON will pick what to do about canonicalization; the JOSE WG is dealing…

Just curious: how shoud signing require canonicalization in any way?
As long as you have a media type and an encoding, the content is fully
defined, right? How would JSON be any different than, say, text/plain
here? AFAIK, text/plain has never required any "canonicalization" of
any kind...

---

From: "Paul C. Bryan" <[email-masked]>
Date: Tue, 19 Feb 2013 17:32:54 -0800

Because in some cases, there's no guarantee that the JSON value being
verified will be transmitted identically as an octet-string.
Canonicalization, in theory, allows it to be represented during
signature verification the same way it was when it was originally
signed. I've seen some (ugly) ways of working around this by
base64-encoding the entire JSON value and putting it in a JSON string.

Paul

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 02:39:42 +0100

> verified will be transmitted identically as an octet-string.

Given the amount of discussion generated by this topic alone, it seems
that these "some cases" are more than "some". What would be such a
case?

> Canonicalization, in theory, allows it to be represented during signature
> verification the same way it was when it was originally signed. I've seen
> some (ugly) ways of working around this by base64-encoding the entire JSON
> value and putting it in a JSON string.

Ick. OK, well, the answer seems pretty simple: UTF-8. It is already
recommended, I guess making is a MUST in RFC 4627bis will solve the
issue without even the need for c14n...

Nothing to lose, everything to gain.

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 03:04:54 +0100

On Wed, Feb 20, 2013 at 2:39 AM, Francis Galiegue <[email-masked]> wrote:
[...]
> Nothing to lose, everything to gain.

Some more thoughts about this...

Signing does not require c14n at all. Signing is totally unaware of
the media type. It will sign whatever it is fed with, and does not
care about what this content actually is. So, signing, say:

----
{ "a": "b", "c": "d" }
----

will not yield the same result as signing:

----
{
    "a": "b",
    "c": "d"
}
----

and that is pretty much normal. The scope of action as far as signing
content is concerned is byte-to-byte representation. And yes, this
means that UTF-16 in these two examples will yield a different
signature than UTF-8. OK, but whatever encoding is used, it will be
_known by the receiving end_. As such, signing in itself completely
precludes the need for c14n, and as such, c14n is not needed at all
since signing is not mandatory.

What _is_ required is that interpretations on either point of a
point-to-point transaction involving JSON data interpret this data in
the same way. And as such, the more pressing matter is turning the
SHOULD NOT have duplicate member names in JSON Objects into a MUST
NOT.

But c14n is, basically, unwarranted.

At least, this is my view on the matter.
-- 
Francis Galiegue, [email-masked]
Try out your JSON Schemas: http://json-schema-validator.herokuapp.com

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2013 18:19:22 -0800

I was careful to call the document in my straw man "comparison" because that is what is needed. Canonicalization is a step before comparison. It is not needed at any other time.

I propose that we let the people who care most about signature comparison, the JOSE WG, deal with whether or not they need a canonicalization method. If they do, they can ask us; if they don't, we will have saved a huge amount of time.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 20:36:15 -0600

> care about what this content actually is. So, signing, say:

Yes and no.  If the verifier and the signer both have the same
document then no c14n is needed.  If the verifier must reconstruct the
signed document -as opposed to receiving it from the signer- then the
verifier must reconstruct exactly the signed document or the signature
will not verify.

Now, when does the latter come up?  Rarely.  You could say that
application protocols must not work that way, but this seems imprudent
to me.  Instead the right thing to do is either define JSON c14n now
or say that we don't need JSON c14n right now and won't define it
until we do.

Nico
--

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 20 Feb 2013 04:00:59 +0100

On Wed, Feb 20, 2013 at 3:36 AM, Nico Williams <[email-masked]> wrote:
[...]
> verifier must reconstruct exactly the signed document or the signature

There is one thing I don't get: in any case, what is transmitted over
the network is just a stream of bytes. One end writes that stream, the
other reads it.

In order for the receiving end to interpret that data, should signing
be used, it needs to verify that the _byte stream_, not its
interpretation, is correct. That byte stream MAY be JSON. It may not
be.

The problem is therefore, IIUC, that if the protocol exchange has
determined that this byte stream is JSON, then there should exist a
canonical form. OK, but then what about this:

{
    "x": 0.00000000000000000000000000000000000000000000000000000000000000000000000000002
}

This is valid JSON. However hard you wish to canonicalize it, the fact
is, some languages won't read it correctly -- in fact, most languages
won't.

And c14n won't help here.

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 21:11:42 -0600

> the network is just a stream of bytes. One end writes that stream, the

No, in this one case the two ends construct some data.  A good example
would be channel bindings (RFCs 5056, 5929), except that mostly that
has no structure, so it's not really a good example after all, but it
illustrates the point.

> interpretation, is correct. That byte stream MAY be JSON. It may not

That's just it: in this case the data isn't transmitted, only the
signature.  There's many protocols that transmit signatures (or
hashes) but not necessarily contents.  E.g., rsync.  What if you had a
JSON-based synchronization protocol and you're sending file metadata,
only there's a lot of it (e.g., large ACLs), and you're trying to
avoid sending it, so you send file names and metadata hashes, and if
the receiver's don't match then you send the actual metadata?

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Tue, 19 Feb 2013 19:18:09 -0800

OK, this discussion has convinced me that there’s no real need for this
group to proactively take up JSON c14n.  If at some future point there’s a
strong demonstrated real (not hypothetical) use case, it’s a fairly
tractable problem.  But for now, it’s unnecessary work.

-T

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 21:21:56 -0600

> tractable problem.  But for now, it’s unnecessary work.

Yup, I think that's the consensus.

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 20 Feb 2013 14:49:42 +1100

> OK, this discussion has convinced me that there’s no real need for this group to proactively take up JSON c14n.  If at some future point there’s a strong demonstrated real (not hypothetical) use case, it’s a fairly tractable problem.  But for now, it’s unnecessary work.

There is interest in JSON canonicalization, despite the trepidation: draft-staykov-hu-json-canonical-form; this email discussion; http://wiki.laptop.org/go/Canonical_JSON; discussions in JOSE WG; http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object; a couple of github projects; bencode.

Add to that the fact that it was felt necessary to define c14n for XML and ASN.1 BER (DER and CER).

C14n has caused lots of pain so a JSON c14n spec should have a big warning up front to use c14n sparingly, only where other protocol choices are worse.

Various JSON c14n proposals so far agree on the basics (sort objects keys, no extra white space, 1 form for numbers), but skimp on some fine details (which chars to escape, escape as \uxxxx or \x, floating point numbers). A spec to fix those details for interoperability seems like a good thing to do at the IETF; now.

draft-staykov-hu-json-canonical-form is 4 pages (including boilerplate, examples, everything). It needs a few more to nail down the details.

I recommend adding JSON c14n to the charter.

--
James Manger

From: [email-masked] [mailto:[email-masked]] On Behalf Of Tim Bray
Sent: Wednesday, 20 February 2013 2:18 PM
To: Nico Williams
Cc: Francis Galiegue; Paul Hoffman; Paul C. Bryan; [email-masked]
Subject: Re: [Json] Canonicalization

OK, this discussion has convinced me that there’s no real need for this group to proactively take up JSON c14n.  If at some future point there’s a strong demonstrated real (not hypothetical) use case, it’s a fairly tractable problem.  But for now, it’s unnecessary work.
-T

---

From: Riccardo Bernardini <[email-masked]>
Date: Wed, 20 Feb 2013 08:13:01 +0100

On Wed, Feb 20, 2013 at 2:32 AM, Paul C. Bryan <[email-masked]> wrote:

> **

Moreover, without canonalization an attack (described, for example, in
"Applied Cryptography," Bruce Schneier, Wiley) is possible.

Briefly, suppose that Eve, who can ask Bob to sign a "good" JSON, wants to
send to Alice a "bad" JSON signed by Bob.  In order to do that, Eve takes a
"good" JSON object that Bob is willing to sign and the "bad" JSON object
that she wants to send to Alice and does some random semantically
negligible changes (e.g., adding spaces, newlines or reordering entries) to
both objects until she finds two versions that hash to the same value.  By
the birthday paradox this will take approximately 2^(b/2) trials, where b
is the hash size. Now she can ask Bob to sign the good object and "stick"
Bob's signature on the bad object.

Of course, there are many ways to counteract such an attack.  For example,
one can use an hash with b=256 bits, so that 2^128 is still a very large
number;  Bob could do some semantically negligible changes himself before
signing or add a random "nonce" somewhere.  Nevertheless, this example
suggests that the possibility of having several semantically equivalent
descriptions of the same object could give rise to security issues.

Riccardo

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 20 Feb 2013 09:29:17 +0000

>Or I have completely misundertood the point...

It depends why you are canonicalizing.  If it's for crypto reasons (for
which I think c14n is usually bad fit), then you're trying to ensure that
different implementations achieved the same octet sequence from the same
starting point.  Therefore, you want to ensure that you only ever generate
exactly one of "a" or "\u0061".

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 20 Feb 2013 14:05:03 +0100

On Feb 20, 2013, at 02:21, Paul Hoffman <[email-masked]> wrote:

> Canonicalizing is part of the signature process

I hope not.

I want to use JSON in constrained systems.

Anything that is saddled with a process as complex as canonicalization will turn out to be a non-starter for this area of application.

If there is a value in (1) taking apart (parsing) a JSON text, (2) keeping the parse tree (or some other form, which by the way is not standardized in any way), then (3) reconstructing a JSON text from that non-standard form, and then (4) checking a digest/authentication value from the reconstructed text, I'd like to know about that benefit.  Not in theoretical terms, but in terms of building actual systems.

By the way, I do find it theoretically interesting to be able to compare two JSON texts at a semantic level.
(Having that equality relation defined amounts to having defined that semantics.  That might help.*)

But please don't do any useful spec that requires implementing c14n.
(Or, at least do another equivalent spec without c14n that makes the previous one unnecessary to implement.)

And, if you actually do need to do c14n for something, do it by turning the JSON text into a byte string that is optimized for being the result of c14n, not into JSON text again.
(You can start from msgpack if you need help in visualizing such a byte string.  Probably much simpler than msgpack, though.)

Grüße, Carsten

*)
{"foo": 1, "bar": 2} =?= {"bar": 2, "foo": 1}
[1234123412341234.1] =?= [1234123412341234.2]
[12341234123412341234.1] =?= [12341234123412341234.2]
[1234123412341234] =?= [1234123412341234.0]
etc.
Just to make these interesting, I posit that all of these pairs are not equal.
This is obviously wrong for anyone but me, but which of the other 15 alternatives is righter?  Why?

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 21 Feb 2013 09:58:09 +1100

Canonicalization (c14n) may be a dirty word, and barred from the charter, but I hope it isn't too inappropriate to still discuss it on this email list.

There is a better c14n format for numbers than I or draft-staykov-hu-json-canonical-form have suggested: it is the format produced by JSON.stringify, as defined in ECMAScript v5.1 [http://www.ecma-international.org/ecma-262/5.1/#sec-15.12.3]. Examples: 0, 1, -1000, 0.00002334, 1500000, 6.022e+23, 9.1e-28.

In fact, c14n for JSON can (and should) be simply defined as per JSON.stringify, with two extra constraints: object name/value pairs are sorted; and \uxxxx escapes use lowercase hex digits (a-f). Done. Developers even have a readily available reference implementations simply by typing JSON.stringify(...) into the JavaScript console of their browser.

--
James Manger

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Manger, James H
> Sent: Wednesday, 20 February 2013 11:58 AM
> To: [email-masked]
> We should avoid the need to canonicalize JSON whenever possible, but
> there are enough efforts to define a canonical form that it would be
> worth standardizing one.
> 
> Strings:
> The simplest string canonicalization rule would be to escape those 34
> chars and no others. A rule that might be slightly nicer for people
> (and hence worth the few extra lines of code) would be to always use
> the 7 \x escapes for those 7 chars, always use \uxxxx for the rest of
> %x00-1F, and never use it for any other chars. I would be tempted to
> drop \/ from the list of seven (only in the canonicalization rules),
> because / is so common in URIs etc.
> 
> Objects:
> Presumably sorting occurs on the logical strings, not the (canonically)
> encoded versions. So {"a\"b":1,"a#b":2} is in canonical form ("(U+22) <
> #(U+23) < \(U+5C)).
> Presumably sorting uses Unicode code points, not UTF-16 words, or UTF-8
> bytes. So {"\uFFE0\uFFE1":3,"\uD834\uDD1E":4} is in the right order
> (0xFFE0 < 0x01D11E), thought the canonical form would use UTF-8 not
> \uxxxx for the 3 characters.
> 0, 1, 1e3, 2.334e-5, 1.5e6 are the sort of canonical form numbers
> should have. I don’t like 0.0E0 for zero as per draft-staykov-hu-json-
> canonical-form-00 -- a person would never write that. That draft also
> allows any number of trailing 0’s (eg 1.200000e-2), which is a bug. A
> canonical form should drop the exponent when it is zero, and drop the
> decimal point when there is nothing after it.

---

From: Francis Galiegue <[email-masked]>
Date: Thu, 21 Feb 2013 00:05:50 +0100

> In fact, c14n for JSON can (and should) be simply defined as per JSON.stringify, with two extra constraints: object name/value pairs are sorted; and \uxxxx escapes use lowercase hex digits (a-f). D…

Not done. By saying that it should be done as per JSON.stringify(),
you basically preclude the use of JSON for languages which _do_ have
the ability to represent numbers outside of IEEE 754, and that
includes numbers as simple as 0.1.

It is not because ECMA 262 has made the braindead decision of limiting
numbers to IEEE 754 that JSON should follow way. RFC 4627 didn't make
that mistake and I certainly hope any IETF-related work concerning
JSON will not make such a mistake either.

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 21 Feb 2013 10:20:02 +1100

> > Canonicalization (c14n) may be a dirty word, and barred from the
> charter, but I hope it isn't too inappropriate to still discuss it on
> this email list.
> >
> > There is a better c14n format for numbers than I or draft-staykov-hu-
> json-canonical-form have suggested: it is the format produced by
> JSON.stringify, as defined in ECMAScript v5.1 [http://www.ecma-
> international.org/ecma-262/5.1/#sec-15.12.3]. Examples: 0, 1, -1000,
> 0.00002334, 1500000, 6.022e+23, 9.1e-28.
> >
> > In fact, c14n for JSON can (and should) be simply defined as per
> JSON.stringify, with two extra constraints: object name/value pairs are
> sorted; and \uxxxx escapes use lowercase hex digits (a-f). Done.
> Developers even have a readily available reference implementations
> simply by typing JSON.stringify(...) into the JavaScript console of
> their browser.

> Not done. By saying that it should be done as per JSON.stringify(), you
> basically preclude the use of JSON for languages which _do_ have the
> ability to represent numbers outside of IEEE 754, and that includes
> numbers as simple as 0.1.
> 
> that mistake and I certainly hope any IETF-related work concerning JSON
> will not make such a mistake either.

Done (almost).

The JSON.stringify format for numbers is NOT limited to 64-bit floats. It effectively says format the number without an exponent if it is in the range [1e-6, 1e21); otherwise format it with 1 digit before the decimal point. There is no limit to the number of digits, ie the precision.

--
James Manger

---

From: Francis Galiegue <[email-masked]>
Date: Thu, 21 Feb 2013 00:31:51 +0100

On Thu, Feb 21, 2013 at 12:20 AM, Manger, James H
<[email-masked]> wrote:
[...]
> The JSON.stringify format for numbers is NOT limited to 64-bit floats. It effectively says format the number without an exponent if it is in the range [1e-6, 1e21); otherwise format it with 1 digit…

Well, I can tell you that it _is_ limited...

Go to the site in my signature, try this:

{
    "maximum": 209098901232398109823091823.2983908091823098098019283019823098,
    "exclusiveMaximum": true
}

as a schema, and as data, input:

209098901232398109823091823.29839080918230980980192830198230981

As I use a language which has the capability to deal with such
numbers, this answers false -- but I use JSON.stringify() to format
the results of validation: look at how the numbers are formatted.

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 21 Feb 2013 11:33:52 +1100

Francis,
Your number is greater than 1e21 so the c14n format would be:
2.090989012323981098230918232983908091823098098019283019823098e+26

If you are exchanging data with an implementation that rounds numbers to fit in 64-bit floats then you obviously will not agree on the exact value. I don't think that is really a c14n issue, however.

Perhaps an IETF JSON c14n spec would need to repeat the JSON.stringify rules as opposed to just referencing them. My point is that JSON.stringify picks a really useful form for numbers (and strings), it is a canonical form (exactly 1 form for any number or string), it is already precisely specified, there are multiple production implementations. That is a great basis for a highly-interoperable IETF JSON c14n spec.

--
James Manger

> From: Francis Galiegue [mailto:[email-masked]]
> Sent: Thursday, 21 February 2013 10:32 AM
> To: Manger, James H
> Cc: [email-masked]; [email-masked]
> > The JSON.stringify format for numbers is NOT limited to 64-bit
> floats. It effectively says format the number without an exponent if it
> is in the range [1e-6, 1e21); otherwise format it with 1 digit before
> the decimal point. There is no limit to the number of digits, ie the
> precision.
> >
> 
>     "maximum":
> 209098901232398109823091823.2983908091823098098019283019823098,
>     "exclusiveMaximum": true
> }
> 
> As I use a language which has the capability to deal with such numbers,
> this answers false -- but I use JSON.stringify() to format the results
> of validation: look at how the numbers are formatted.
> 
> --

---

From: Nico Williams <[email-masked]>
Date: Tue, 19 Feb 2013 14:59:14 -0600

> I also haven't seen a clear use case that canonicalization solves that can't be more easily solved another way.

But what were the mistakes?  Is canonicalization in and of itself a
mistake?  Or were the mistakes about how to do it?

I personally think that standards should avoid having to canonicalize
JSON (and XML) wherever possible.  The simplest way to do this is
never re-encode in the process of validating signatures, for example.
However, I'm not sure that we can always avoid canonicalization, or
that even if we can it has no value; I could be convinced.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Feb 2013 14:11:07 +0100

>> or at least know that it happened?

Well, I did engage the msgpack community some more.

You can find a transcript of some 275 messages about separating byte and UTF-8 strings at:

	https://github.com/msgpack/msgpack/issues/121

Summary:
Some members of the msgpack community are very enraged that this change hasn't happened earlier.
Of course, some have gone off and done their own incompatible forks.
Others are very enraged that any change is happening at all, and that new people are intruding on their turf.
(And some probably feel guilty that it took a ****storm from outside to finally make this change.)

frsyuki is now working on a proposal that solves the problem:

	https://gist.github.com/frsyuki/5022569

The proposal is technically complete (and has already been implemented).
It already is pretty good at the details, too, but this whole thing is being done in a process that is closer to Japanese consensus processes than to IETF culture.

My -01 will be fully aligned with whatever the state of frsyuki's proposal will be on Monday's I-D deadline (find today's snapshot at http://www.tzi.de/~cabo/draft-bormann-apparea-bpack-01pre2.txt).
(frsyuki's proposal may change some more, but those will in all likelihood be minor details.)
I think his overall thinking is fine, but it is much more dominated by a requirement for backwards compatibility than an IETF process would be.

So, the larger question on whether the msgpack community is ready to take part (or just endure) in an IETF-style consensus process (including handing over change control) still looms.

That doesn't diminish from the requirement for a msgpack-like format, and I think we should use Hallway Time in Orlando to discuss potential ways forward.

I any case, I definitely don't want to disturb the constructive discussion about chartering a very narrow JSON fixup WG with this work.
(I do want to find a home for it, soon, though: I want to build other specs on top of it.)

Grüße, Carsten