Subject: [Json] Consensus call: establishing name equality

From: Paul Hoffman <[email-masked]>
Date: Fri, 21 Jun 2013 09:42:07 -0700

There are four proposals for establishing name equality:

0) Leave the current draft as-is, not discussing name equality

1) In Section 2.5 ("Strings"), immediately before the ABNF add:
   For purpose of establishing name equality, comparisons MUST be conducted, after all unescaping
   is done, by comparing numeric character code points. There is to be no modification of any
   kind to the characters in names, including case-changing or combining-form normalization.
   For example, the following four names MUST be considered equivalent:
    * "\u002F"
    * "\u002f"
    * "\/"
    * "/"

2) In Section 2.5 ("Strings"), immediately before the ABNF add:
   For purpose of establishing name equality, comparisons MUST be conducted, after all unescaping
   is done, by comparing numeric character code points. There MUST NOT be any modification of any
   kind to the characters in names, including change of case or change between precomposed and
   decomposed forms.
   For example, the following four names MUST be considered equivalent:
    * "\u002F"
    * "\u002f"
    * "\/"
    * "/"

3) In Section 2.5 ("Strings"), immediately before the ABNF add:
   For purpose of establishing name equality, implementations MUST first do all unescaping and
   then MUST compare numeric character code points. There is to be no modification of any kind to
   the characters in names, including case-changing or combining-form normalization.
   For example, the following four names MUST be considered equivalent:
    * "\u002F"
    * "\u002f"
    * "\/"
    * "/"

Please respond to this message with a list of proposals you could accept, ordered from highest to lowest. Do not list proposals you cannot live with. If you cannot accept any of the proposals, please respond and say why.

Based on the responses we receive, we will try to judge the consensus of the WG.

-- The JSON WG co-chairs

---

From: Tim Bray <[email-masked]>
Date: Fri, 21 Jun 2013 10:02:01 -0700

#1 and #3 only differ at the editorial-nit level; I’m equally fine with
either.  -T

On Fri, Jun 21, 2013 at 9:42 AM, Paul Hoffman <[email-masked]> wrote:

> Please respond to this message with a list of proposals you could accept,
> ordered from highest to lowest. Do not list proposals you cannot live with.
> If you cannot accept any of the proposals, please respond and say why.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Stephan Beal <[email-masked]>
Date: Fri, 21 Jun 2013 19:21:08 +0200

> #1 and #3 only differ at the editorial-nit level; I’m equally fine with

+1

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Jun 2013 20:18:16 +0200

Hmm, a bit too much MUSTard for my taste.  Only the first MUST in 1 is really needed.
So 1 > 3 > 2, but I don't like any of them:
a) The example does not illustrate the sentence preceding it (although it sounds as if it were intended to do so):  Swap the example and the "no mods" sentence.
(Again, b) we are comparing sequences of code points and not single code points, and c) the "there is to be no mod" sentence is not properly bound to the comparing, so there is infinite potential for misinterpretation of these sentences unless you already know what they are saying.)

Do we have consensus what we'll use the name equality for that we just defined?

Grüße, Carsten

---

From: "Jim Schaad" <[email-masked]>
Date: Fri, 21 Jun 2013 14:04:58 -0700

3
1
2

Jim

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Paul Hoffman
> Sent: Friday, June 21, 2013 9:42 AM
> To: JSON WG
> Subject: [Json] Consensus call: establishing name equality
> 
>    kind to the characters in names, including case-changing or
combining-form
> normalization.
>    For purpose of establishing name equality, implementations MUST first
do
> all unescaping and
> ordered from highest to lowest. Do not list proposals you cannot live
with. If
> you cannot accept any of the proposals, please respond and say why.
> 
> Based on the responses we receive, we will try to judge the consensus of
the
> WG.

---

From: Jacob Davies <[email-masked]>
Date: Fri, 21 Jun 2013 15:23:15 -0700

> Do we have consensus what we'll use the name equality for that we just

That's my question. What is the problem this is seeking to solve again?
Because I'm inclined to say 0, especially since keys cannot be guaranteed
to be unique (and so no definition of uniqueness needs to be defined for
that). It seems like this is dictating the interpretation of data, rather
than its format.

---

From: Tim Bray <[email-masked]>
Date: Fri, 21 Jun 2013 15:29:37 -0700

Hm, I think this is useful.  I’ve seen people go off the rails trying to be
overly clever comparing strings, so this in effect is just telling them not
to overthink. -T

---

From: Nico Williams <[email-masked]>
Date: Fri, 21 Jun 2013 17:31:22 -0500

> overly clever comparing strings, so this in effect is just telling them not

Is that yet another instance of breakage we're not allowed to cause?

---

From: Jacob Davies <[email-masked]>
Date: Fri, 21 Jun 2013 16:15:20 -0700

On Fri, Jun 21, 2013 at 3:29 PM, Tim Bray <[email-masked]> wrote:

> Hm, I think this is useful.  I’ve seen people go off the rails trying to
> be overly clever comparing strings, so this in effect is just telling them
> not to overthink. -T
>

Is this definition of equality needed elsewhere in the document though?
Standing alone it seems a little odd - there's no discussion how to compare
two numbers, two objects, or two lists for equality, so why are
strings-used-for-names special?

---

From: Tim Bray <[email-masked]>
Date: Fri, 21 Jun 2013 16:21:53 -0700

> strings-used-for-names special?

Because
- We talk about duplicates so it's important to define what duplicate means
- they are, in effect, hash-table keys, so comparison really matters

---

From: Vinny A <[email-masked]>
Date: Fri, 21 Jun 2013 18:43:17 -0500

On Jun 21, 2013, at 6:15 PM, Jacob Davies <[email-masked]> wrote:
> Hm, I think this is useful.  I’ve seen people go off the rails trying to be overly clever comparing strings, so this in effect is just telling them not to overthink. -T
> 
> Is this definition of equality needed elsewhere in the document though? Standing alone it seems a little odd - there's no discussion how to compare two numbers, two objects, or two lists for equality, so why are strings-used-for-names special?

View this as more of a constraint on names, not on strings in general. It's important to set expectations here, as implementations have to convert JSON into maps.

> Please respond to this message with a list of proposals you could accept, ordered from highest to lowest. Do not list proposals you cannot live with. If you cannot accept any of the proposals, plea…

3 = 1 > 2

I can't live with option 0, as I think there needs to be more clarification in regards to names, name comparison, and conversion to language-specific maps.

-----------------
Vinny
www.jsontest.com

---

From: Norbert Lindenberg <[email-masked]>
Date: Fri, 21 Jun 2013 16:45:20 -0700

> - We talk about duplicates so it's important to define what duplicate means

Correct - section 2.2, "The names within an object SHOULD be unique."

> - they are, in effect, hash-table keys, so comparison really matters

That seems out of scope for the spec - the spec may cover the output of the parser, but not what clients do with the output. We can't stop clients from treating names as case insensitive, for example.

However, the comment that eventually led to this proposal was actually about a somewhat different problem [1]:

> It should say explicitly that these three all produce exactly the same result:
>     "\u002F"
>     "\/"
>     "/"
> 
> I have seen confusion about this.

This comment is about how the parser interprets JSON text and what its output is. The three input strings shown, as well as "\u002f", should all result in an output string containing the single code point U+002F.

This important information seems to have gotten lost in the subsequent discussion.

To say that the four input strings must be considered equivalent then becomes unnecessary because they result in the same name.

[1] http://www.ietf.org/mail-archive/web/json/current/msg00372.html

Norbert

---

From: Jacob Davies <[email-masked]>
Date: Fri, 21 Jun 2013 16:54:47 -0700

Well, even given those givens, what about this:

http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf
>From Section 3.2:

C6 A process shall not assume that the interpretations of two
canonical-equivalent character sequences are distinct.

• The implications of this conformance clause are twofold. First, a
process is never required to give different interpretations to two
different, but canonical-equivalent character sequences. Second, no
process can assume that another process will make a distinction
between two different, but canonical-equivalent character sequences.

• Ideally, an implementation would always interpret two
canonical-equivalent character sequences identically. There are
practical circumstances under which implementations may reasonably
distinguish them.

---

That argues pretty strongly against specifying name normalization
behavior for implementations, I would think.

Back on the point of what this is for: yes, JSON names in objects are
likely used as hash keys. But the specification itself does not
*require* that they be unique (it advises, yes) and therefore no
absolutely unambiguous definition of what "unique" means is necessary
within the specification.

The specification is already clear that a name is a string, and that
backslashed characters in JSON-encoded strings are to be interpreted
in certain ways to produce a native string; how to hash and compare
strings is really a platform function, not a decoding function. It
seems unnecessary to say that the four given strings for the slash
character are equivalent: if you have followed the instructions on
interpreting strings, they already will be by the time you get around
to comparing them.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 21 Jun 2013 17:14:59 -0700

<chair hat on>

On Jun 21, 2013, at 3:23 PM, Jacob Davies <[email-masked]> wrote:

> What is the problem this is seeking to solve again?

Some people want to compare names for objects. The current draft says:
   The names within an object SHOULD be unique.
If either a generator or parsers wants to follow that rule (as some do), some people in the WG feel that there needs to be a standard way to do so. This call is to determine how to do that, or whether to try.

--Paul Hoffman

---

From: Jacob Davies <[email-masked]>
Date: Fri, 21 Jun 2013 17:58:25 -0700

On Fri, Jun 21, 2013 at 5:50 PM, John Cowan <[email-masked]> wrote:
> JSON, like XML, does not assume they are distinct: it decides that they
> are distinct.  This is what in Unicode is called a higher-order protocol.

Hm, OK.

> SHOULD is rather more than advice.  Quoth RFC 2119: "This word, or
> the adjective "RECOMMENDED", mean that there may exist valid reasons
> in particular circumstances to ignore a particular item, but the full
> implications must be understood and carefully weighed before choosing
> a different course."
>
> If JSON document, generator, or parser authors choose to ignore this
> stipulation, they need to know what it is they are choosing not to do.

Okey dokes.

Then: 2, 3, 1, 0

---

From: John Cowan <[email-masked]>
Date: Fri, 21 Jun 2013 20:50:14 -0400

-- 
Is not a patron, my Lord [Chesterfield],        John Cowan
one who looks with unconcern on a man           http://www.ccil.org/~cowan
struggling for life in the water, and when      [email-masked]
he has reached ground encumbers him with help?
        --Samuel Johnson

---

From: Stefan Drees <[email-masked]>
Date: Sat, 22 Jun 2013 15:30:05 +0200

On 2013-06-21 18:42 +02:00, Paul Hoffman wrote:
> Please respond to this message with a list of proposals you could
> accept, ordered from highest to lowest. Do not list proposals you cannot
> live with. If you cannot accept any of the proposals, please respond and
> say why. ...

[3,2,1]

plus: I am happily hopping on John's train, trying to leave "numeric 
character code points" town for some better place ...

{"Stefan":true}

---

From: R S <[email-masked]>
Date: Sat, 22 Jun 2013 09:18:50 -0700

[0]

On Sat, Jun 22, 2013 at 6:30 AM, Stefan Drees <[email-masked]> wrote:

>> combining-form normalization.
>> do all unescaping and
> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 24 Jun 2013 15:18:04 +0900

> are distinct.  This is what in Unicode is called a higher-order protocol.

This is just playing with words. In the general use of the words, I'd 
think that "decide" is a specific(ally narrow) version of "assume". And 
just as a higher-order protocol would be extremely ill-advised to treat 
upper-case and lower-case characters as distinct in a situation where 
the two get converted into each other all the time, the same applies 
here. I don't remember having seen the phrase "higher-order protocol" 
anywhere near or in relationship to C6; if you have, I'd appreciate you 
telling me where.

The truth of the matter is that C6, to a large extent, is based on 
wishful thinking: Canonically equivalent strings are one and the same, 
and software may and will convert from one normalization form to another 
at almost every corner, so you're never sure which one you get, and 
better don't bet on getting one or the other.

In actual practice, the kinds of software that changes character 
normalization has turned out to be few and far between. The MacIntosh 
file system is (one of) the exception(s) that proves the rule. In 
particular, there are no known instances of changes of character 
normalization during network transit, which is what's most important for 
us here.

Regards,   Martin.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Jun 2013 10:15:29 +0200

> canonical-equivalent character sequences are distinct.

I think that this argument suffers a bit from not distinguishing a string and its representation.

JSON is never "interpreting" strings, so this clause doesn't seem to apply.

It *is* processing the representation of a string in order to derive the actual string from that.
E.g., by unescaping the character sequences in that representation into the characters of the string, and by removing the enclosing quotes etc.  It may indeed be worth pointing out that this processing is not "interpreting" the Unicode character sequences either *).

(On of the persistent problems with IETF specifications is that the only formal description method used widely, ABNF, makes the reader focus on the representation, when actually it would often be more useful to focus on the represented data object.  It is almost always crucial to clearly distinguish the two.  Clearly, what we are interchanging are the represented objects, so some purists argue we never need to even talk about their interpretation.  That is a fallacy, because there is no interchange unless the interpretations match well enough.)

I think the discussion here would benefit a lot from always clearly making this distinction.
"a" is not a three-character string, it is a three-character JSON representation of a single-character string.

Grüße, Carsten

*): How do you like "\u028ä"?  Well, that may seem invalid and is a bit hard to read, but if may mail client hadn't normalized it under my fingers, it would be the same as "\u028a\u0308", which is a credible upsilon diaeresis (ʊ̈, always a two-character string as there is no combined form).  (In building JSON for interchange, it is probably not so bright to assume it will never be normalized in transit.  At least if you assume that JSON should survive being sent around in e-mail messages.  But that observation is firmly in best practices land...)

---

From: John Cowan <[email-masked]>
Date: Mon, 24 Jun 2013 12:52:32 -0400

Carsten Bormann scripsit:

> (On of the persistent problems with IETF specifications is that the only
> formal description method used widely, ABNF, makes the reader focus
> on the representation, when actually it would often be more useful to
> focus on the represented data object.  It is almost always crucial to
> clearly distinguish the two.

Indeed it is!

> Clearly, what we are interchanging are the represented objects,

I would say just the opposite: clearly, what we are exchanging is the
representations.  The represented objects never go anywhere: they are
re-created at the destination and are hopefully isomorphic to those
that existed at the source.

I'm not sure if you really disagree or it was just a lapsus linguae.

> so some purists argue we never need to even talk about their
> interpretation.  That is a fallacy, because there is no interchange
> unless the interpretations match well enough.)

Indeed.

> *): How do you like "\u028ä"?  Well, that may seem invalid and is a
> bit hard to read, but if may mail client hadn't normalized it under my
> fingers,

The moral here is never to place a combining character directly after
markup, or it will combine with the markup.  The W3C Character Model
draft makes this point.

> it would be the same as "\u028a\u0308", which is a credible
> upsilon diaeresis 

Not so much; Latin upsilon (used in IPA) has serifs, which makes it
quite un-Greek in form.  The proper upsilon with diaeresis (dialytika)
is U+03CB.

-- 
John Cowan   http://ccil.org/~cowan  [email-masked]
[P]olice in many lands are now complaining that local arrestees are insisting
on having their Miranda rights read to them, just like perps in American TV
cop shows.  When it's explained to them that they are in a different country,
where those rights do not exist, they become outraged.  --Neal Stephenson

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Jun 2013 21:06:05 +0200

Hey John,

it is dangerous to write to a mailing list where people cite the previous writer in Latin :-)

On Jun 24, 2013, at 18:52, John Cowan <[email-masked]> wrote:

> representations.  

We completely agree.
I was just using "represented" as a short form of "converted to a representation".
To repeat, we are not interchanging the objects, but representations of them.

>> a credible

I know that I should have added "IPA" to this... :-)
(Of course, it just was a constructed example.  
I have never seen this particular problem in actual interchange.)

Grüße, Carsten