Subject: [apps-discuss] #3: json-pointer escape characters

From: Mark Nottingham <[email-masked]>
Date: Wed, 27 Jun 2012 12:32:50 +1000

This is <http://trac.tools.ietf.org/wg/appsawg/trac/ticket/3>.

The current draft uses ^, and Paul explains why the draft chose that over %-encoding.

I haven't seen any response to Paul's message. Julian (and Joe, who also supported this), are you OK with leaving it as-is?

Regards,

On 13/04/2012, at 2:21 AM, Paul C. Bryan wrote:

> This takes us back to a previous version of JSON Pointer, which in fact did use URI escaping of %2F to distinguish between the '/' token delimiter and a '/' character within a token. IIRC, the objections that lead us to change were:
> 
> 1. It requires URI decoding logic in non-URI-processing code (e.g. when pointers are used in JSON strings rather than in URI fragment identifiers). This has always been a somewhat weak argument, IMO.
> 
> 2. Some URI parsers normalize the URI (i.e. expand %2F into '/') before any JSON Pointer parsing code can get a chance to see it.
> 
> In response to this, I went down the path of using a single escape character. If we can overcome these objections, I'd be happy to entertain returning to URI escaping.
> 
> Paul
> 
> On Thu, 2012-04-12 at 17:34 +0200, Julian Reschke wrote:
>> Hi there,
>> 
>> right now, the draft escapes forward slashes in reference tokens using 
>> ^-escapes, that is:
>> 
>>   a/b
>> 
>> becomes
>> 
>>   a^/b
>> 
>> and
>> 
>>   a^b
>> 
>> becomes
>> 
>>   a^^b
>> 
>> (Reminder: previously "\" was used, but it results in ugly double 
>> escaping when the pointer occurs in a JSON string).
>> 
>> A drawback of this scheme is that when the pointer is used inside a URI, 
>> such as the path component of a an HTTP URI, the / still needs to be 
>> escaped, so the name
>> 
>>    a/b
>> 
>> becomes the pointer
>> 
>>    a^/b
>> 
>> and would end up as
>> 
>>    a%5e%2fb
>> 
>> in the URI.
>> 
>> 
>> An alternate proposal I heard during the IETF meeting in Paris was to 
>> simply apply URI percent escaping to the characters in the URI 
>> gen-delims range:
>> 
>>   gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"
>> 
>> so
>> 
>>    a/b
>> 
>> would simply become
>> 
>>    a%2fb
>> 
>> and wouldn't need any further escaping in a URI path component.
>> 
>> Feedback appreciated,
>> 
>> Julian
>> _______________________________________________
>> apps-discuss mailing list
>> 
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/apps-discuss
> 
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Jun 2012 09:19:41 +0200

On 2012-04-12 18:21, Paul C. Bryan wrote:
> This takes us back to a previous version of JSON Pointer
> <http://tools.ietf.org/html/draft-pbryan-zyp-json-pointer-02>, which in
> fact did use URI escaping of %2F to distinguish between the '/' token
> delimiter and a '/' character within a token. IIRC, the objections that
> lead us to change were:
>
> 1. It requires URI decoding logic in non-URI-processing code (e.g. when
> pointers are used in JSON strings rather than in URI fragment
> identifiers). This has always been a somewhat weak argument, IMO.

The way you state is slightly misleading. You don't need URI decoding 
logic; you just need percent-decoding logic (which is only small part of 
URI parsing). And you need *some* kind of unescaping anyway, so I don't 
see the material difference...

> 2. Some URI parsers normalize the URI (i.e. expand %2F into '/') before
> any JSON Pointer parsing code can get a chance to see it.

Again; don't use a URI parser. Just %-unescape.

> In response to this, I went down the path of using a single escape
> character. If we can overcome these objections, I'd be happy to
> entertain returning to URI escaping.

I think it would be worthwhile to do.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 27 Jun 2012 20:39:36 +1000

> Again; don't use a URI parser. Just %-unescape.

I thought the point here was that if it appeared in a URI, it would be prone to being escaped at the wrong time.

Unless you're talking about double-percent-escaping, which I really hope you're not.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Jun 2012 13:33:42 +0200

> Unless you're talking about double-percent-escaping, which I really hope you're not.

No, I wasn't; I misunderstood the issue.

So the problem is about *preserving* the escaped slash when processing 
the URI.

Test:

- a JSON name of "a/b"

- the JSON pointer would be "a%2Fb"

- embedded into a URI it might show up as: "http://example.com/a%2Fb"

- parsing this might loose the escaping too early, returning a hier-part 
(http://greenbytes.de/tech/webdav/rfc3986.html#components) of "/a/b" 
instead of "/a%2AFb"?

This may indeed be a problem.

Best regards, Julian

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 27 Jun 2012 01:13:21 -0600

I'll say that I wish that we had found a way to use %-escaping for XEP-106
(http://xmpp.org/extensions/xep-0106.html), and that I wish I could go back
in time and talk then-me out of caring about the edge case that drove us
there.

I think argument #1 is irrelevant; implementing %-escaping from scratch is
easy.

Argument #2 makes sense, but one could specify that in URI contexts, you
%-encode '%' as %25, so to match a literal '/', you specify %252F in
URI-land.

Regardless, if everyone else likes ^, I'll still use JSON-pointer, and will
merely shake my fist at the heavens a little every time I have to implement
another encoder and decoder in a new programming language.

On 6/26/12 8:32 PM, "Mark Nottingham" <[email-masked]> wrote:

> I haven't seen any response to Paul's message. Julian (and Joe, who also
> supported this), are you OK with leaving it as-is?
> 
> Regards,
> 
> 
> 
>> This takes us back to a previous version of JSON Pointer, which in fact did
>> use URI escaping of %2F to distinguish between the '/' token delimiter and a
>> '/' character within a token. IIRC, the objections that lead us to change
>> were:
>> 
>> pointers are used in JSON strings rather than in URI fragment identifiers).
>> This has always been a somewhat weak argument, IMO.
>> 
>> 2. Some URI parsers normalize the URI (i.e. expand %2F into '/') before any
>> JSON Pointer parsing code can get a chance to see it.
>> 
>> In response to this, I went down the path of using a single escape character.
>> If we can overcome these objections, I'd be happy to entertain returning to
>> URI escaping.
>> 
>> Paul
>> 

-- 
Joe Hildebrand

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 28 Jun 2012 14:21:48 +1000

The "best" syntax for a JSON pointer is a JSON array: no new escaping; and references to an object element or array item are easily distinguished (using a string or a number respectively). This is suitable for carrying a pointer in a JSON document. Examples:

  ["foo",0]
  ["a/b"]
  ["i\\j"]

It is pretty awful for putting a pointer into a URI fragment, however.

  #%5B%22foo%22,0%5B

A syntax with one layer of %-escaping is theoretically possible since a URI fragment allows / and %2F as separate things. However, it does clash with APIs. For instance, the java.net.URI(scheme, host, path, fragment) constructor will not let you generate …#/a%2Fb. Pass in "/a%2Fb" and you get "…#/a%252Fb".

Double %-escaping would work (1st layer in pointer code, 2nd layer in URI fragment code). It's just that double escaping seems very likely to confuse developers (mind you, all escaping confuses). Would it be sufficient for the pointer code just to do replaceAll("%2[fF]","/").replaceAll("%25","%") or would it have to cope with arbitrary other bytes being unnecessarily escaped?

A quick web search revealed half-a-dozen implementations [1-6] in various languages of an early version of the JSON pointer spec that used %-escapes. They all split the pointer on '/', then decode each segment.

This simple approach no longer works with the current JSON pointer spec. split("/") fails as a "/" may be a delimiter, but may also be part of an "^/" escape sequence. To match only delimiters requires a regular expression with a potentially unlimited amount of "look-behind" to see if there are an odd number of preceding carets: (?<![^^](?:\^\^)*\^)/. Java (v1.6), for example, does not support unlimited look-behind. (?<![^^](?:\^\^){0,99}\^)/ is an approximation that does work.

Switching to ~0 and ~1 as escapes for ~ and / would allow split("/") to work, followed by replaceAll("~1","/").replaceAll("~0","~"). Creating a pointer is also easy: replaceAll("~", "~0").replaceAll("/", "~1") then join the parts. "~" also doesn't need to be %-escaped in a URI. I assume most languages have split() and replaceAll() functions.

[1] https://github.com/janl/node-jsonpointer
[2] https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py
[3] https://github.com/raphaelstolt/php-jsonpointer/blob/master/src/JsonPointer/JsonPointer.php
[4] http://sources.forgerock.org/browse/~raw,r=303/commons/json-fluent/trunk/src/main/java/org/forgerock/json/fluent/JsonPointer.java
[5] http://pydoc.net/jsonpointer/0.1/jsonpointer
[6] …

--
James Manger

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 28 Jun 2012 08:37:59 -0700

After my initial visceral reaction against this suggestion, I've come
around to really like it.

+1

Paul

On Thu, 2012-06-28 at 14:21 +1000, Manger, James H wrote:

> The "best" syntax for a JSON pointer is a JSON array: no new escaping;
> and references to an object element or array item are easily
> distinguished (using a string or a number respectively). This is
> suitable for carrying a pointer in a JSON document. Examples:
> 
>   ["foo",0]
>   ["a/b"]
>   ["i\\j"]
> 
> A syntax with one layer of %-escaping is theoretically possible since
> a URI fragment allows / and %2F as separate things. However, it does
> clash with APIs. For instance, the java.net.URI(scheme, host, path,
> fragment) constructor will not let you generate …#/a%2Fb. Pass in "/a%
> 2Fb" and you get "…#/a%252Fb".
> 
> 
> Double %-escaping would work (1st layer in pointer code, 2nd layer in
> URI fragment code). It's just that double escaping seems very likely
> to confuse developers (mind you, all escaping confuses). Would it be
> sufficient for the pointer code just to do replaceAll("%
> 2[fF]","/").replaceAll("%25","%") or would it have to cope with
> arbitrary other bytes being unnecessarily escaped?
> 
> 
> A quick web search revealed half-a-dozen implementations [1-6] in
> various languages of an early version of the JSON pointer spec that
> used %-escapes. They all split the pointer on '/', then decode each
> segment.
> 
> This simple approach no longer works with the current JSON pointer
> spec. split("/") fails as a "/" may be a delimiter, but may also be
> part of an "^/" escape sequence. To match only delimiters requires a
> regular expression with a potentially unlimited amount of
> "look-behind" to see if there are an odd number of preceding carets:
> (?<![^^](?:\^\^)*\^)/. Java (v1.6), for example, does not support
> unlimited look-behind. (?<![^^](?:\^\^){0,99}\^)/ is an approximation
> that does work.
> 
> 
> Switching to ~0 and ~1 as escapes for ~ and / would allow split("/")
> to work, followed by replaceAll("~1","/").replaceAll("~0","~").
> Creating a pointer is also easy: replaceAll("~", "~0").replaceAll("/",
> "~1") then join the parts. "~" also doesn't need to be %-escaped in a
> URI. I assume most languages have split() and replaceAll() functions.
> 
> 
> [2]
> https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py
> [3]
> https://github.com/raphaelstolt/php-jsonpointer/blob/master/src/JsonPointer/JsonPointer.php
> [4]
> http://sources.forgerock.org/browse/~raw,r=303/commons/json-fluent/trunk/src/main/java/org/forgerock/json/fluent/JsonPointer.java

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 29 Jun 2012 07:44:45 +1000

Let's see how it looks in the examples we just inserted.

{
   "foo": ["bar", "baz"],
   "": 0
   "a/b": 1,
   "c%d": 2,
   "e^f": 3,
   "g|h": 4,
   "i\\j": 5,
   "k\"l": 6,
   " ": 7
}

In JSON strings:
        
 ""         // the whole document
 "/foo"       ["bar", "baz"]
 "/foo/0"    "bar"
 "/"          0
 "/a~1b"      1
 "/c%d"       2
 "/e^f"      3
 "/g|h"       4
 "/i\\j"      5
 "/k\"l"      6
 " "          7

and in URIs fragments:

 #                  // the whole document
 #/foo            ["bar", "baz"]
 #/foo/0         "bar"
 #/               0
 #/a~1b         1
 #/c%25d          2
 #/e%5Ef       3
 #/g%7Ch          4
 #/i%5Cj          5
 #/k%22l          6
 #/%20            7

Of course, we don't have any ~ examples, but it does seem cleaner in both encodings.

On 29/06/2012, at 1:37 AM, Paul C. Bryan wrote:

> After my initial visceral reaction against this suggestion, I've come around to really like it.
> 
> +1
> 
> Paul
>

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Jun 2012 15:31:03 -0700

> Of course, we don't have any ~ examples, but it does seem cleaner in both encodings.

As a quoting character, ~ is a good choice.  I might offer that 0 and
1 are strange choices for the next character, but I wont offer an
alternative except to note that double-escape is the more commonly
used method for escaping the escape character.

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 28 Jun 2012 15:36:09 -0700

True, ~~ would just as good to escape ~ instead of ~0...

Paul

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 29 Jun 2012 10:19:49 +1000

> True, ~~ would just as good to escape ~ instead of ~0...

No.

Using "~~" to escape "~" means a "~" in a pointer can be
either an escaper or an escapee -- and you have to count
a potentially indefinite number of preceding "~"'s to
determine the answer. That just makes parsing and matching
with a regex harder.

The names in the following 2 pointers end in "1" or "/".
Which is which?

 /~~~~~~~~~~~~~~~~~~~~1
 /~~~~~~~~~~~~~~~~~~~1

It is much more obvious if we use "~0" as the escape for "~".

 /~0~0~0~0~0~0~0~0~0~01
 /~0~0~0~0~0~0~0~0~0~1

--
James Manger

---

From: Edward Cree <[email-masked]>
Date: Fri, 29 Jun 2012 01:56:06 +0100

On 29/06/12 01:19, Manger, James H wrote:
>>> 1 are strange choices for the next character
> determine the answer.
0 and 1 are still strange choices, though; it might be better to use 
'descriptive' single letters, perhaps ~s for a slash and either ~t for 
't'ilde or ~e for 'e'scape.
However, "~t" might be a confusing choice as "\t" usually means 't'ab.

Of course, a regex is in any case the Wrong way to unescape a string, 
but it's not our job to tell implementers what to do.
--
Edward Cree

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 29 Jun 2012 11:21:13 +1000

Edward,

> but it's not our job to tell implementers what to do.

Why is using a regular expression the Wrong way to unescape a string?
Is it because the simplest implementation uses a separate pass over the whole string for each escape sequence, which is inefficient?

Suggesting a Right way to implementers is part of the job of a standard.

--
James Manger

---

From: Edward Cree <[email-masked]>
Date: Fri, 29 Jun 2012 03:31:32 +0100

> Is it because the simplest implementation uses a separate pass over the whole string for each escape sequence, which is inefficient?
No, it's because a regular expression engine is needless complexity, as 
well as its troubles with things like the "~~" scheme.
IMHO the Right way to unescape (or, for that matter, escape) a string is 
to run a state machine along it.  This is efficient, it can handle many 
escape sequences in a single pass, the set of escapes can be encoded 
once in a table and used by both the escaping and unescaping functions 
(thus preserving SPOT) which is hard to do with a regex based approach.
But do enlighten me if I'm missing something.

To clarify, by "run a state machine along it" I mean something like this:

    char c;
    bool esc=false;
    while((c=inputString.getCharacter()))
    {
         if(esc)
         {
             switch(c)
             {
                 case '~'
                     outputString.appendCharacter('~');
                 break;
                 case '1'
                     outputString.appendCharacter('/');
                 break;
                 default:
                     outputString.appendCharacter(c);
                 break;
             }
             esc=false;
         }
         else if(c=='~')
             esc=true;
         else
             outputString.appendCharacter(c);
    }

This has no problem dealing with "~~~~~~~~~~~~~~~~~~~~~~~~1", because it 
doesn't need to backtrack.  It's also likely to have less overhead than 
a regex.
--
Edward Cree

---

From: William Mills <[email-masked]>
Date: Fri, 29 Jun 2012 08:01:51 -0700 (PDT)

I agree with Edward's later message, and specifically about what he's calling backtracking.  Unescaping SHOULD be a single pass through the sting, and specifically a an escape character rendered into the result string should never then be evaluated in a new escape sequence.  The example I would choose here is HTML, "The &amp; character may be represented in HTML as &amp;amp;.".  If you follow the implicit logic of your question you would end up with "The & character may be represented in HTML as &.", which is wrong.  

-bill

>________________________________
> From: "Manger, James H" <[email-masked]>
>To: Paul C. Bryan <[email-masked]>; Martin Thomson <[email-masked]> 
>Cc: Mark Nottingham <[email-masked]>; IETF Apps Discuss <[email-masked]>; Joe Hildebrand <[email-masked]> 
>Sent: Thursday, June 28, 2012 5:19 PM
>Subject: Re: [apps-discuss] #3: json-pointer escape characters
>

---

From: "Paul C. Bryan" <[email-masked]>
Date: Fri, 29 Jun 2012 08:44:30 -0700

I wouldn't say wrong, but I would say inefficient.

Paul

---

From: "Paul C. Bryan" <[email-masked]>
Date: Fri, 29 Jun 2012 08:46:15 -0700

I'll concede that ~0 is easier for humans and regular expression parsers
to process than ~~, so I'll retract any suggestion that it should be so.