Subject: [Json] Strings

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 13:19:24 -0700

It should say explicitly that these three all produce exactly the same 
result:

     "\u002F"
     "\/"
     "/"

I have seen confusion about this.

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 5 Jun 2013 22:41:54 +0200

> I have seen confusion about this.

Me too. By default, PHP e.g. serializes a "/" to "\/". I'm wondering whether
it would make sense to recommend that serializers use "/"?

--
Markus Lanthaler
@markuslanthaler

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 14:29:30 -0700

> it would make sense to recommend that serializers use "/"?
PHP does that because it usually generates HTML. If you embed JSON on 
HTML, then it is sometimes necessary to escape /, so that strings like 
"</script>" don't cause disaster. Strings like "<\/script>" are safe. 
PHP could be smarter about how it does that, but it is not my job to 
tell PHP how to be smarter.

---

From: Stephan Beal <[email-masked]>
Date: Wed, 5 Jun 2013 23:33:17 +0200

On Wed, Jun 5, 2013 at 11:29 PM, Douglas Crockford <[email-masked]>wrote:

> Strings like "<\/script>" are safe. PHP could be smarter about how it does
> that, but it is not my job to tell PHP how to be smarter.
>

They recently (5.4) added a flag to disable that:

http://php.net/manual/de/function.json-encode.php

See JSON_UNESCAPED_SLASHES.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 17:14:21 -0700

<no hat>

On Jun 5, 2013, at 1:19 PM, Douglas Crockford <[email-masked]> wrote:

> It should say explicitly that these three all produce exactly the same result:
> 
>    "\u002F"
>    "\/"
>    "/"

Proposed wording: just before the ABNF at the end of section 2.5, add:

When comparing strings, escaped characters SHOULD compare the same as their unescaped counterparts. For example, the following three strings SHOULD be considered equivalent for comparisons:

   "\u002F"
   "\/"
   "/"

--Paul Hoffman

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 6 Jun 2013 10:32:52 +1000

> When comparing strings, escaped characters SHOULD compare the same as
> their unescaped counterparts. For example, the following three strings
> SHOULD be considered equivalent for comparisons:
> 
>    "\u002F"
>    "\/"
>    "/"

Surely these MUST compare the same, not SHOULD. They represent exactly the same string (sequence of code points).

Also include "\u002f" (lowercase hex) as another form that produces exactly the same result.

P.S. Another little fix: update reference to ABNF from RFC4234 to RFC5234.

--
James Manger

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 18:22:49 -0700

> P.S. Another little fix: update reference to ABNF from RFC4234 to RFC5234.

Sounds good to me, as long as a few people promise to do a careful checking of that.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 18:23:36 -0700

On Jun 5, 2013, at 5:32 PM, "Manger, James H" <[email-masked]> wrote:

> Surely these MUST compare the same, not SHOULD.

Not for a new requirement in this document, no.

> They represent exactly the same string (sequence of code points).

Yes.

> Also include "\u002f" (lowercase hex) as another form that produces exactly the same result.

Good addition.

--Paul Hoffman

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 18:57:07 -0700

> Sounds good to me, as long as a few people promise to do a careful checking of that.
Why does this matter? Does RFC5234 offer us any better precision?

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 19:00:28 -0700

> Why does this matter? Does RFC5234 offer us any better precision?

No, but in the IETF, when one RFC is updated by another, there is an expectation that a new RFC will always refer to the latter.

--Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 03:35:43 +0000

>SHOULD be considered equivalent for comparisons:

Are you sure you want to open pandora's box of comparing strings?

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Wed, 5 Jun 2013 22:51:36 -0500

> Are you sure you want to open pandora's box of comparing strings?

Object key lookup is of comparable complexity to string comparison,
and rather fundamental to any use of JSON.  So, yes, we may have to
say something.  Like that all-ASCII, all-printable, non-backslash keys
have a single canonical form, and then leave the rest unspecified.

I wish Unicode normalization were cheap and code for it plentiful.
The latter is only just starting to be the case.  Well,
normalization-insensitive comparison/hashing of mostly-ASCII strings
can actually be quite fast.

BTW, stedolan.github.io/jq is a really nifty JSON *filter* language (a
bit of XPath and XSLT, but for JSON, so highly simplified).
Unsurprisingly jq lets users do JSON value comparisons, including
string comparisons.  Also unsurprisingly, it has no concept of Unicode
normalization.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 21:41:14 -0700

> Are you sure you want to open pandora's box of comparing strings?

+1. I don’t think we need go there. -T

>
> --
> Joe Hildebrand
>
>
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Jun 2013 08:33:28 +0200

> Why does this matter? Does RFC5234 offer us any better precision?

Minor bugfixes in the way things have been written up, no technical changes:

http://tools.ietf.org/rfcdiff?url2=rfc5234&url1=rfc4234

(Updating is good spec hygiene, and the RFC editor will make you update it anyway.  Or the IESG before it.)

Grüße, Carsten

---

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 03:53:56 -0700

> No, but in the IETF, when one RFC is updated by another, there is an expectation that a new RFC will always refer to the latter.
Who will assure that no problems will be introduce if we do this?

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 07:59:20 -0700

> Are you sure you want to open pandora's box of comparing strings?

Yes. String comparison is used for object keys, which is a hot topic in another thread.

{ "a": 1, "\u0061": 2 }

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 08:32:03 -0700

> Who will assure that no problems will be introduce if we do this?

Many people on the mailing list, hopefully.

Will one or more of you who care about good ABNF volunteer to do careful checks against RFC 5234?

--Paul Hoffman

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Thu, 6 Jun 2013 16:43:00 +0000

> Yes. String comparison is used for object keys, which is a hot topic in another thread.

To (probably vainly) help keep a lid on the box, do we need to note there is no expectation of normalization?

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Jun 2013 18:46:55 +0200

> Will one or more of you who care about good ABNF volunteer to do careful checks against RFC 5234?

Well, so far I only checked that 5234 is not worse for 4627bis than 4234.
But I can also volunteer to torture-test the ABNF itself.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Thu, 6 Jun 2013 09:48:49 -0700

If keys are unicode strings, the proper languagee is:

“For purpose of establishing key equality, comparisons MUST be conducted,
after all escaping is done, on a character-by-character basis by comparing
numeric character codepoints. There is to be no modification of any kind to
the characters in keys, including case-changing or combining-form
normalization.”

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 09:52:08 -0700

> To (probably vainly) help keep a lid on the box, do we need to note there is no expectation of normalization?

We do *not* need to note that, because we can't define it in under a page of text.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 09:54:10 -0700

> But I can also volunteer to torture-test the ABNF itself.

Thank you, we have a volunteer! Who will be the second? That's a serious question: some WGs have passed non-ABNF-compliant documents due to having too few checkers. Fortunately, there is little ABNF in this document, so that should cause others to want to gain this new and valuable skill.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 6 Jun 2013 18:59:38 +0200

On Jun 6, 2013, at 18:54, Paul Hoffman <[email-masked]> wrote:

> some WGs have passed non-ABNF-compliant documents due to having too few checkers.

I got this wrong once, and the IETF chair had this message for me:

http://www.youtube.com/watch?feature=player_embedded&v=LbK-g8tKnoc

Yes, we want to get this right.
(And sorry for invoking Godwin's law so early.)

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 17:51:20 +0000

On 6/6/13 10:48 AM, "Tim Bray" <[email-masked]> wrote:

>³For purpose of establishing key equality, comparisons MUST be conducted,
>after all escaping is done, on a character-by-character basis by
>comparing numeric character codepoints. There is to be no modification of
>any kind to the characters in keys, including
> case-changing or combining-form normalization.²

maybe remove character-by-character as redundant and possibly confusing?

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 6 Jun 2013 17:53:02 +0000

On 6/6/13 10:54 AM, "Paul Hoffman" <[email-masked]> wrote:

>Thank you, we have a volunteer! Who will be the second? That's a serious
>question: some WGs have passed non-ABNF-compliant documents due to having
>too few checkers. Fortunately, there is little ABNF in this document, so
>that should cause others to want to gain this new and valuable skill.

I've got some adequate abnf parsing code that I wrote last year against
5234.  I'll at least run that.

-- 
Joe Hildebrand

---

From: Vinny A <[email-masked]>
Date: Thu, 6 Jun 2013 17:44:41 -0500

On Jun 6, 2013, at 11:54 AM, Paul Hoffman <[email-masked]> wrote:
> Thank you, we have a volunteer! Who will be the second? That's a serious question: some WGs have passed non-ABNF-compliant documents due to having too few checkers.

I will also volunteer, if you're looking for more.

Thanks.
-V

-----------------
Vinny A
www.jsontest.com

---

From: Nico Williams <[email-masked]>
Date: Thu, 6 Jun 2013 17:59:29 -0500

> after all escaping is done, on a character-by-character basis by comparing

and after unescaping unnecessarily-escaped characters.

> the characters in keys, including case-changing or combining-form

This works, and stronly implies (as I would like to have it) that the
only reliably comparable strings (for the purposes of object keys
[names]) are those where normalization to any normalization form would
have no effect on the codepoints making up the string.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 01:02:11 +0000

On 6/6/13 4:59 PM, "Nico Williams" <[email-masked]> wrote:

>> normalization.²

Why is that implied?  We're not suggesting normalizing names in any way,
other than unescaping.

-- 
Joe Hildebrand

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 11:03:53 +1000

> “For purpose of establishing key equality, comparisons MUST be conducted, after all escaping is done, on a character-by-character basis by comparing numeric character codepoints. There is to be no modification of any kind to the characters in keys, including case-changing or combining-form normalization.”

s/after all escaping is done/after all unescaping is done/

or should that be 

s/after all escaping is done/after all escaping is undone/

--
James Manger

---

From: Tim Bray <[email-masked]>
Date: Thu, 6 Jun 2013 19:05:55 -0700

+1 to either, I had it backward

On Thu, Jun 6, 2013 at 6:03 PM, Manger, James H <
[email-masked]> wrote:

> > “For purpose of establishing key equality, comparisons MUST be
> conducted, after all escaping is done, on a character-by-character basis by
> any kind to the characters in keys, including case-changing or
> combining-form normalization.”
>
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 07 Jun 2013 18:16:09 +0900

On 2013/06/07 1:52, Paul Hoffman wrote:
> On Jun 6, 2013, at 9:43 AM, Matt Miller (mamille2)<[email-masked]>  wrote:
>

We might need more than a page to describe how we wanted to normalize 
strings if we wanted to normalize them. But to say that there is *no* 
such expectation doesn't take much more than a few lines of text, as 
show by Tim.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 07 Jun 2013 18:28:23 +0900

> +1 to either, I had it backward

I have a very minimal preference for the second variant, but I'm happy 
with either.

> On Thu, Jun 6, 2013 at 6:03 PM, Manger, James H<

While we are at it, please change "codepoint" to "code point". That's 
how Unicode spells it.

I'm not exactly happy with the "there is to be no" part. I guess that 
was an attempt at avoiding a MUST or MUST NOT. I don't think that's 
necessary, we are already on a normative level. Also, I'd prefer to 
avoid long nouns. So what about:

"There MUST NOT be any modification of any kind to the characters in 
keys, including change of case or change between precomposed and 
decomposed forms.".

or if we want to avoid another normative UPPER CASE, then something like

"Modifications of any kind to the characters in keys, including change 
of case or change between precomposed and decomposed forms."

A more correct way to express the last part ("change between precomposed 
and decomposed forms") would be "or change between canonically 
equivalents or compatibility equivalents [UAX #15]",
where UAX #15 is http://www.unicode.org/reports/tr15/. That's because 
Unicode normalization includes things such as singletons, which have 
nothing to do with a precomposed/decomposed choice.

Regards,   Martin.