Subject: [apps-discuss] Feedback on draft-ietf-appsawg-json-pointer-00

From: Mike Acar <[email-masked]>
Date: Thu, 1 Mar 2012 12:14:29 -0800

Hi,

I've been asked to review the JSON Pointer and Patch draft RFCs. I've 
had some discussion about them off-list, and have been advised to bring 
my issues to the list. And so, I am. :)

I've scanned the archived list discussion of the patch and pointer RFCs, 
so I don't think these issues have come up here already. Let me know if 
they have.

The first issue is comparing Unicode strings. Section 4 of 
http://tools.ietf.org/html/draft-ietf-appsawg-json-pointer-00 says

       If the currently referenced value is a JSON object, the new
       referenced value is the object member with the name identified by
       the reference token.

That is, if the reference token equals the name of some value within the 
object, move to that value. However, the tokens and values are Unicode 
strings; I'm not an expert in Unicode, but my impression is that testing 
Unicode strings for equality is not as simple as comparing sequences of 
bytes. For example, there are linguistic considerations: I believe 
German ö and oe are considered identical.

There's also the question of JSON documents with different encodings; 
UTF8 is the default, but UTF-16 and -32 with both endiannesses are also 
supported. Presumably this question will disappear in practice, since 
implementations will operate on deserialized data structures, not on 
JSON texts.

Could somebody with more expertise in this area comment? Perhaps this is 
all a solved problem, and the Pointer RFC simply needs to pick a 
normalization method and equality definition - or perhaps it's all so 
well-settled I'm seeing a problem where there is none.

Another issue is that the evaluation scheme implies some application 
semantics. Again, section 4:

    If a reference token is being evaluated against a concrete JSON
    document, the implementation MAY evaluate each token against a
    concrete value, and terminate evaluation with an error condition if a
    evaluation fails to resolve a concrete value.

It's easy to imagine a case where a Pointer refers to a deep hierarchy, 
e.g. /a/b/c/d, and the application semantics of following it are to 
create automatically any non-existent intermediate objects.

My initial thought was that the Pointer RFC should explicitly say that 
specs which use it (e.g. Patch) should define semantics for ambiguous 
cases (non-unique or non-existent member names, array index out of 
bounds, etc). That can still result in a user of Pointer having to 
implement it himself, if there are no implementations which offer him 
precisely what he needs.

Is it preferrable to explicitly say "You must make these decisions for 
your use case" instead of "Behavior in these cases is undefined"?

-- 
Mike Acar -                                 - macar at cloudmark dot com

---

From: Tim Bray <[email-masked]>
Date: Thu, 1 Mar 2012 12:30:04 -0800

On Thu, Mar 1, 2012 at 12:14 PM, Mike Acar <[email-masked]> wrote:

> However, the tokens and values are Unicode
> bytes. For example, there are linguistic considerations...

There are lots of considerations, the W3C once wrote a big huge
complicated spec on normalization forms.  I would say explicitly don’t
go there.  I could see the spec advising implementors to watch out for
this, but two strings are equal if they have the same number of
Unicode characters and the codepoints are positionwise equal;
otherwise not.  -T

---

From: Julian Reschke <[email-masked]>
Date: Thu, 01 Mar 2012 21:30:08 +0100

> German ö and oe are considered identical.

No, they aren't.

The spec could be made a bit clearer by saying "code point by code 
point", but I would think that's the default unless otherwise stated.

> implementations will operate on deserialized data structures, not on

Indeed.

> ...

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 01 Mar 2012 13:45:50 -0800

> German ö and oe are considered identical.

While we may consider ö and oe to be linguistically equivalent, I do no
believe they are considered lexicographically equivalent in a Unicode
string comparison. Someone please correct me if I'm wrong. Would it help
to define the comparison as being lexicographical?

> implementations will operate on deserialized data structures, not on 

Since they're logically the same underlying Unicode representations, I'm
not sure there's any issue to consider here.

> your use case" instead of "Behavior in these cases is undefined"?

I'm not sure. I'm open to suggestions from the apps working group...

Paul

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 02 Mar 2012 12:57:47 +0900

> to define the comparison as being lexicographical?

No. Lexicographical is usually used with respect to order, not 
equivalence (see e.g. http://en.wikipedia.org/wiki/Lexicographical_order).

> not sure there's any issue to consider here.

The best way to spec this is to say that equivalence is checked 
codepoint-by-codepoint. That solves both issues, because codepoints are 
independent of UTF-8/UTF-16/UTF-32, simply the Unicode character numbers.

Regards,    Martin.

---

From: Tim Bray <[email-masked]>
Date: Thu, 1 Mar 2012 21:16:23 -0800

This thread has convinced me that it’s still easy after all these
years for people to go down rabbit holes on this issue, and therefore
the draft should be explicitly clear that key equality MUST be
determined codepoint-by-codepoint as Martin suggests.

 -T

On Thu, Mar 1, 2012 at 7:57 PM, "Martin J. Dürst"
<[email-masked]> wrote:
> No. Lexicographical is usually used with respect to order, not equivalence
> (see e.g. http://en.wikipedia.org/wiki/Lexicographical_order).
>
>
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: "Paul C. Bryan" <[email-masked]>
Date: Fri, 02 Mar 2012 10:41:44 -0800

Okay, I will amend the draft accordingly.

Paul

---

From: Mike Acar <[email-masked]>
Date: Fri, 2 Mar 2012 14:19:22 -0800

On 03/01/2012 12:30 PM, Tim Bray wrote:
> On Thu, Mar 1, 2012 at 12:14 PM, Mike Acar<[email-masked]>  wrote:
>
> complicated spec on normalization forms.

I feared this was the case.

> I would say explicitly don’t go there.  I could see the spec advising
> implementors to watch out for this, but two strings are equal if they
> have the same number of Unicode characters and the codepoints are
> positionwise equal; otherwise not.

This sounds good to me.

---

From: Mike Acar <[email-masked]>
Date: Fri, 2 Mar 2012 14:34:33 -0800

> not sure there's any issue to consider here.

Probably not in practice; as you and I discussed off-list, most 
implementations will not actually be working on the JSON texts themselves.

>> semantics.

>> bounds, etc).

There is another case: What to do if a non-terminal reference token 
names a value which is not a structured type; e.g.

{ "a" : "b" }

How should you interpret "/a/c/d"? For Patch this should be an error, 
but another application might promote the value of "a" from a string to 
an object or array. If the Pointer RFC says this MAY be an error, it 
disallows those semantics.

>> That can still result in a user of Pointer having to implement it
>> himself, if there are no implementations which offer him precisely
>> what he needs.
>>
>> Is it preferrable to explicitly say "You must make these decisions
>> for your use case" instead of "Behavior in these cases is
>> undefined"?
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 2 Mar 2012 22:47:48 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Acar
> Sent: Friday, March 02, 2012 2:35 PM
> To: [email-masked]
> Subject: Re: [apps-discuss] Feedback on draft-ietf-appsawg-json-pointer-00
> 
> >> It's easy to imagine a case where a Pointer refers to a deep
> >> hierarchy, e.g. /a/b/c/d, and the application semantics of following
> >> it are to create automatically any non-existent intermediate objects.
> >>
> >> My initial thought was that the Pointer RFC should explicitly say
> >> that specs which use it (e.g. Patch) should define semantics for
> >> ambiguous cases (non-unique or non-existent member names, array index
> >> out of bounds, etc).
> 

I don't agree that MAY disallows anything, but I do think the current expression is too mushy.

I think it's fine for the pointer document to say that the handling of such cases is application-specific, but it should actually say that, perhaps by using your two examples to illustrate how different applications using the pointer specification could behave.

-MSK