Subject: Re: [Jsonpath] Expression language -- index expressions and filter expressions

From: Daniel P <[email-masked]>
Date: Tue, 9 Mar 2021 09:19:49 -0500

> From: Carsten Bormann <[email-masked]>

> I think what we have so far is:
>
> — there needs to be an expression language that is independent from specific
> implementations and their implementation languages.
> — the expression language is used both in index expressions [()] and filter
> expressions [?()].
>
> Obviously, in index expressions we need the expression to return an index value
> (integer, string), while in filter expressions we need a Boolean value.
> Also, in index expressions, the implicit parameter @ is the item to be indexed (the
> item to which the [()] is applied), while in filter expressions, @ is the item to be filtered
> (one of the items in the container to which the [?()] is applied).
>
> We have a dial we can turn how conversion-happy the expression language should
> be.
> E.g., the conversion from the filter expression result to a Boolean can use a big set of > “falsy” values (false, 0, “”, [], {}) or be more restrictive in what it accepts.
> Similarly, the conversion from the index expression result to the array index or map
> key (JSON object member name) could be loose (e.g., converting numbers into
> strings) or more strict.
>
> Generally, both kinds of expressions won’t know statically to which kind of container
> (array, JSON object) they are applied.
> It should be possible to write expressions that can operate with both kinds.
> Surprising and error prone conversions should be avoided, though.
>
It seems to me there are a number of distinct issues when talking
about expressions.

(1) The unary and binary operators to support, and their precedence
and associativity. My own suggested list would be

https://github.com/danielaparker/jsoncons/blob/master/doc/ref/jsonpath/jsonpath.md#filter-predicates

(2) What kinds of operands to support. My suggestions would be (a)
JSONPath expressions, relative to either the root $ or the current
node @, (b) single quoted strings, and (c) JSON literals.

(3) The meaning of an expression with a binary operator and two
operands. In particular, the meaning of an expression when one operand
is a JSONPath expression,
returning say [0], and the other is a JSON literal, say 0 or [0].

(4) As Carsten mentioned, the rules for determining the effective
boolean value of a JSON value.

(5) Functions. Functions are unquestionably useful in filter
expressions. Implementations that use Javascript or another scripting
language for expression evaluation of course have them. There is also
prior experience that can be
drawn on from implementations with their own expression evaluation that
support functions.

Another issue is whether expression evaluation should be a separate
component, or integrated into the JSONPath grammar proper. My view is
the latter. A number of implementations take that approach.

Daniel

---

From: Greg Dennis <[email-masked]>
Date: Tue, 9 Mar 2021 18:35:08 +0000 (UTC)

> Another issue is whether expression evaluation should be a separate component, or integrated into the JSONPath grammar proper. - Daniel
I'm not sure I understand your meaning being this distinction.
Greg 
 
  On Wed, 10 Mar 2021 at 3:20 am, Daniel P<[email-masked]> wrote:   > From: Carsten Bormann <[email-masked]>

Daniel

-- 
Jsonpath mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/jsonpath

---

From: Greg Dennis <[email-masked]>
Date: Tue, 9 Mar 2021 18:45:53 +0000 (UTC)

Stupid autocorrect.  This wouldn't be an problem on GitHub, where I can edit the comment to fix typos.
I'm not sure I understand your meaning behind* this distinction.

  On Wed, 10 Mar 2021 at 7:35 am, Greg Dennis<gregsdennis=[email-masked]> wrote:   -- 
Jsonpath mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/jsonpath

---

From: Daniel P <[email-masked]>
Date: Tue, 9 Mar 2021 14:35:27 -0500

> I'm not sure I understand your meaning being this distinction.
For an example of filter expression integrated into the JSONPath
grammar proper, see

https://github.com/kevinbirch/kanabo/blob/master/jsonpath.ebnf

I haven't studied the ebnf, it dates to 2013, but it presents a simple JSONPath
grammar that integrates path expressions and filter expressions. It's notable
because, as far as I can tell, few implementations, including my modest
contribution, publish an ebnf (working on that.)

The alternative is to have a separately specified ExpressionEvaluator.
In this case,
the containing parentheses in "[?(<expr>)]" serve as top level delimiters for
the expression, and the root, current node, and expression are handed off to the
ExpressionEvaluator for evaluation.

Note that if the filter expression is integrated into the JSONPath
grammar proper,
such that the JSONPath parser understands the expression syntax, then the
containing parentheses are no longer necessary, we could have "[?<expr>]".

Daniel

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 9 Mar 2021 21:19:56 +0100

> The alternative is to have a separately specified ExpressionEvaluator.

As Greg pointed out, same difference.

(Yes, the *interface* can [and should!] be made well-defined, but whether there is a section break before the ExpressionEvaluator grammar rules begin does not make a difference.)

Grüße, Carsten

---

From: Daniel P <[email-masked]>
Date: Tue, 9 Mar 2021 18:32:44 -0500

On Tue, Mar 9, 2021 at 3:19 PM Carsten Bormann <[email-masked]> wrote:
> (Yes, the *interface* can [and should!] be made well-defined, but whether there is a
> section break before the ExpressionEvaluator grammar rules begin does not make a
> difference.)
>
They can be functionally the same, but there are of course differences
between a single JSONPathEvaluator, and separate JSONPathEvaluator and
ExpressionEvaluator.

>From a spec point of view, the ExpressionEvaluator requires some
duplication in its grammar, for example, it requires unquoted-string
and single-quoted-string. Paths occur in expressions, presumably the
ExpressionEvaluator will not fully understand path syntax, but will
need to know enough to extract them to be evaluated with the
JSONPathEvaluator.

>From a non-functional perspective, it is noted that implementers will
likely follow the grammar, so if the grammar specifies separate
components, implementers will probably implement it that way. This
requires a JSONPathEvaluator and ExpressionEvaluator that invoke each
other, with more memory requirements, and the likelihood of deeper
stack recursion and less performance. Also syntax error checking is
more robust if the parser fully understands the tokens it is parsing,
instead of simply knowing enough to extract a string to hand off to
another component. That's not to say the error wouldn't be detected,
it would, but the diagnostic could be less helpful.

I don't want to make the case too large, but I do think there is a
case to be made for having the filter expression integrated into the
JSONPath grammar proper.

Daniel

---

From: Greg Dennis <[email-masked]>
Date: Wed, 10 Mar 2021 00:02:36 +0000 (UTC)

> They can be functionally the same, but there are of course differences between a single JSONPathEvaluator, and separate JSONPathEvaluator and ExpressionEvaluator.
This is getting into the implementation space, which we shouldn't direct.  Implementations need to be free to define their own architectures.  Our role is to define requirements.
Greg

Daniel

---

From: Daniel P <[email-masked]>
Date: Tue, 9 Mar 2021 19:41:38 -0500

> This is getting into the implementation space, which we shouldn't direct.  Implementations need to be free to define their own architectures.  Our role is to define requirements.
Regardless, the WG still needs to choose whether to integrate the
filter expression grammar into the JSONPath grammar proper, or define
a separate grammar for the filter expression. That's all. My
preference would be for the former. Others may have a different
opinion.

Daniel

---

From: Greg Dennis <[email-masked]>
Date: Wed, 10 Mar 2021 03:43:50 +0000 (UTC)

Daniel, could you explain the two grammars, please? I'm not sure I follow what you're suggesting.  I see only a single overall grammar, where expressions are a subset of that grammar and only valid under certain circumstances.
Surely, the expression grammar will be defined within the same spec (it doesn't make sense to write an entirely new spec for an expression grammar that will only be used in JSON Path), but I can't see expression grammar just being used anywhere within a path.  It only makes sense where an expression is valid.  Additionally, I would expect that the path grammar should be valid _within_ an expression, e.g. `@.name`.
Separately, there exists a question (which has been mentioned on GitHub) regarding nested expressions, which also leads to scoping for `@`, but I think that can be left out of this discussion for now. 

Daniel