Subject: [Jsonpath] Draft minutes, consensus points, and actions from IETF 112

From: James <[email-masked]>
Date: Sun, 14 Nov 2021 12:35:15 +0100

All,

The minutes for our session at IETF 112 are now uploaded to datatracker here:

https://datatracker.ietf.org/doc/minutes-112-jsonpath/

Consensus:

* #124 - Won't be used, but can be added in future if there's real demand
* #127 - Won't be included in the specification, but can be part of extensions
* #119 - In boolean contexts, path converts to false if nodelist is empty, but does not check value(s)
* #117 - Normalization not part of spec, codepoint comparison should be used
* #109 - We acknowledge but won't address this issue
* #95 - This won't be part of spec as it's an API problem but could be included in future

Outstanding action items that were captured from the meeting:

* Draft Status - Tim to make a PR
* #127 - Example queries need to be written
* #124 - Carsten to comment that consensus is to leave it for an extension
* #123 - Discuss use cases to get consensus of undefined vs NaN on list
* #119 - Carsten to make further edits based on consensus
* #109 - Carsten to update the issue
* #70 - Have discussion on list with known options

Please let me know of any omissions or errata.

- J

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 14 Nov 2021 22:14:48 +0100

> * #70 - Have discussion on list with known options

As a reminder, my slides had these options (renumbered for easier reference, and expanded a bit):

1. Select (define) one regular expression flavor
2. Provide a way to plug in regular expressions (of different flavors)
3. No regexps in base RFC (but keep an extension point)

1. further splits into:

1a. Select *a version of* ECMAScript (parsing/searching RE)
1b. Select W3C XSD RE (matching RE)
1c. Build "modest subset" (e.g., iregexp)

Since we don’t have a consensus or even a majority among the implementations, we are free to do the right thing, if we can pull that off.

As you know, I have been exploring 1c.

I submitted an updated version -01 of draft-bormann-jsonpath-iregexp.
As in -00, I’m using W3C XSD RE as a base, as these are actual regular expressions, amenable to implementation techniques that are less susceptible to DoS problems than the Perl/PCRE/ECMAscript dialect.

Apart from character class subtraction and the exact semantics of Multi-Character escapes (\s \d \w etc., outside and inside of character class expressions), W3C XSD RE are pretty much a consensus subset of the various regular expression dialects, except that they are in the form of matching expressions (no anchors needed) instead of parsing expressions.  

I believe the spec should have conversion instructions for implementers that just want to use the regexp engine they happen to have handy.  Because of the consensus subset nature of W3C XSD RE, these instructions are relatively straightforward (copy, and surround by the anchors the target flavor happens to use).

I tried to add conversion instructions for Multi-Character escapes (\s \d \w and \S \D \W, leaving out the \c and \i that nobody except W3C has).
As you can see when looking at the diff, the result is not pretty when it comes to double negation in character classes.  Maybe a pathological case, but a bit of a trap, and maybe not that useful anyway as the W3C semantics is quite different from the PCRE ones.
So I’m leaning towards a -02 that does not have Multi-Character escapes.
(None of the regexps I found in RFCs uses them, and not having them also happens to be pretty much what the json-schema.org drafts seem to converge at.)

Grüße, Carsten

Status:   https://datatracker.ietf.org/doc/draft-bormann-jsonpath-iregexp/
Html:     https://www.ietf.org/archive/id/draft-bormann-jsonpath-iregexp-01.html
Diff:     https://www.ietf.org/rfcdiff?url2=draft-bormann-jsonpath-iregexp-01

---

From: Tim Bray <[email-masked]>
Date: Sun, 14 Nov 2021 13:20:16 -0800

I suspect that we are not the only WG, in the present or the future, who
need to/will need to address this issue.  I suggest that we should take
this to ietf@ or ask the IESG if they have an opinion.  Francesca, do you
have any thoughts?

On Sun, Nov 14, 2021 at 1:15 PM Carsten Bormann <[email-masked]> wrote:

> Since we don’t have a consensus or even a majority among the
> implementations, we are free to do the right thing, if we can pull that off.
>
> As in -00, I’m using W3C XSD RE as a base, as these are actual regular
> expressions, amenable to implementation techniques that are less
> susceptible to DoS problems than the Perl/PCRE/ECMAscript dialect.
>
> Apart from character class subtraction and the exact semantics of
> Multi-Character escapes (\s \d \w etc., outside and inside of character
> class expressions), W3C XSD RE are pretty much a consensus subset of the
> various regular expression dialects, except that they are in the form of
> matching expressions (no anchors needed) instead of parsing expressions.
>
> I believe the spec should have conversion instructions for implementers
> that just want to use the regexp engine they happen to have handy.  Because
> of the consensus subset nature of W3C XSD RE, these instructions are
> relatively straightforward (copy, and surround by the anchors the target
> flavor happens to use).
>
> I tried to add conversion instructions for Multi-Character escapes (\s \d
> \w and \S \D \W, leaving out the \c and \i that nobody except W3C has).
> As you can see when looking at the diff, the result is not pretty when it
> comes to double negation in character classes.  Maybe a pathological case,
> but a bit of a trap, and maybe not that useful anyway as the W3C semantics
> is quite different from the PCRE ones.
> (None of the regexps I found in RFCs uses them, and not having them also
> happens to be pretty much what the json-schema.org drafts seem to
> converge at.)
>
> Grüße, Carsten
>
> Html:
> https://www.ietf.org/archive/id/draft-bormann-jsonpath-iregexp-01.html
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-bormann-jsonpath-iregexp-01
>
> --
> JSONpath mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jsonpath
>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 14 Nov 2021 23:10:52 +0100

On 2021-11-14, at 22:20, Tim Bray <[email-masked]> wrote:
> 
> I suspect that we are not the only WG, in the present or the future, who need to/will need to address this issue.  I suggest that we should take this to ietf@ or ask the IESG if they have an opinion.  

This is not a new subject to the IETF…

The two specifications that already use regexps I am most familiar with are YANG and CDDL (*); both use W3C XSD REs.  Defining a useful subset that is easy to implement (e.g., based on conversion instructions to popular dialects) would benefit both ecosystems.

I’m not sure ietf@ is the right place to bring this up.
dispatch@ is closer (we could also start at art@).
It is probably best to move the discussion over there when we have a reasonably solid proposal; I don’t think asking the question in a vacuum will be productive.

So I think we should first try to do what is good for jsonpath, maybe check next with cbor@ and netmod@ (the people who do YANG) whether we have missed anything that would allow them to just adopt (in the English sense) our result, and then maybe ask at dispatch@ whether they see a problem with completing this spec in the jsonpath WG (which might bring us a few new faces — I don’t think that could hurt).

> Francesca, do you have any thoughts?

Listening attentively...

Grüße, Carsten

(*)
(Of course, there are NAPTRs, which use POSIX ERE for a transformation (replacement) operation.
OpenPGP uses "the same syntax as the Henry Spencer's "almost public domain" regular expression [REGEX] package.”, but then goes on to define a iregexp-like subset with anchors.
Various documents use regular expressions as a specification technique; these usually point to POSIX but then use an iregexp-like subset; e.g., RFC 7970.
RFC 4730 goes as far as defining a specific regexp dialect for dialing digit sequences, DRegEx.
I don’t think any of these are strongly related to the matching regexp intention of JSONpath, CDDL, or YANG.)

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 29 Nov 2021 01:08:21 +0100

I have not forgotten these action points, but I actually first created a pull request based on Michael McCool’s comments:

https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-base/pull/137

I have structured this into multiple commits in case we can’t merge all of this immediately.

New (somewhat) technical content:

* The text is now using I-JSON (RFC 7493) to discuss integer number value ranges.
  This removes some weasel wording that is likely to cause interop issues.
* Some of the text about false/truthy is gone, some more changes will be needed.
* The text about negative indexes is now less misleading.

There is also a little markdown adjustment, and a lot of editorial fixes.

Thanks to Michael for the review!

Grüße, Carsten