Subject: [Jsonpath] truthy and falsy

From: Daniel P <[email-masked]>
Date: Wed, 8 Sep 2021 13:54:07 -0400

Glyn Normington wrote

> The terms "truthy" and "falsy" are used in the current draft, but are not
> defined. Please could someone provide a (pointer to a) good definition?

Perhaps this may be helpful?

https://craftinginterpreters.com/evaluating-expressions.html#truthiness-and-falsiness

Given that the conditions for being "falsey" are somewhat arbitrary,
and that most of them are represented in one implementation or
another, it may be useful to have an appendix that provides an
analysis of some of the differences between the main implementations,
and the rationale for the one settled on.

Daniel

---

From: Glyn Normington <[email-masked]>
Date: Wed, 8 Sep 2021 18:59:51 +0100

Thanks Daniel. That link explained the idea nicely. We also need to settle
on the specifics for JSONPath and document them.

On Wed, 8 Sep 2021, 18:54 Daniel P, <[email-masked]> wrote:

> --
> Jsonpath mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jsonpath
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 Sep 2021 20:16:22 +0200

> https://craftinginterpreters.com/evaluating-expressions.html#truthiness-and-falsiness

As this text states, the concept of falsy is of interest only if there are implicit conversions to Boolean.

What are the places where we want to have those implicit conversions?

Grüße, Carsten

---

From: Daniel P <[email-masked]>
Date: Wed, 8 Sep 2021 14:19:16 -0400

Glyn,

In the jsoncons implementations, I followed the JMESPath rule,
described in https://jmespath.org/specification.html#or-expressions.
Although I have to confess to being drawn to the simpler rule in Ruby
and Lua: where only false and null (nil) are considered false.

Daniel

On Wed, Sep 8, 2021 at 2:00 PM Glyn Normington
<[email-masked]> wrote:
>
> Thanks Daniel. That link explained the idea nicely. We also need to settle on the specifics for JSONPath and document them.
>

---

From: Greg Dennis <[email-masked]>
Date: Wed, 8 Sep 2021 19:41:42 +0000 (UTC)

I want us to be careful about supporting truthiness.  I've already commented on this several times (e.g. https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-base/issues/64#issuecomment-831755108) that it's difficult to support in strongly-typed languages.

Greg
 
  On Thu, 9 Sep 2021 at 6:19 am, Daniel P<[email-masked]> wrote:   Glyn,

Daniel

---

From: Daniel P <[email-masked]>
Date: Wed, 8 Sep 2021 17:01:22 -0400

Dennis,

I don't understand your comment. In your cited comment, you comment on
the binary operators '==' vs '===' in JS, and express a preference for
the latter. But that's a separate issue from "supporting truthiness",
which is about evaluating a _single_ value as a boolean.

Daniel

---

From: Greg Dennis <[email-masked]>
Date: Wed, 8 Sep 2021 21:29:27 +0000 (UTC)

It's all the same in strong languages.  Casting of any kind is difficult.
Greg

  On Thu, 9 Sep 2021 at 9:01 am, Daniel P<[email-masked]> wrote:   Dennis,

Daniel

---

From: Daniel P <[email-masked]>
Date: Wed, 8 Sep 2021 19:13:43 -0400

Greg Dennis <[email-masked]> wrote:

>It's all the same in strong languages.  Casting of any kind is difficult.

Casting? What casting? And what is all "the same"?

Determining whether a single value is truthy or falsey doesn't entail casting or
conversion of any kind. It simply involves noting the value's type,
and comparing
that against a checklist.

Daniel

---

From: Greg Dennis <[email-masked]>
Date: Thu, 9 Sep 2021 00:21:26 +0000 (UTC)

It's converting an arbitrary value to a boolean.  This isn't trivial in strongly-types languages like c# (and other .net languages).
Greg

  On Thu, 9 Sep 2021 at 11:13 am, Daniel P<[email-masked]> wrote:   Greg Dennis <[email-masked]> wrote:

Daniel

---

From: Daniel P <[email-masked]>
Date: Wed, 8 Sep 2021 20:47:51 -0400

Well, the ruby/lua test for truthiness is surely trivial:

 boolean isTruthy(Object object)
{
    if (object == null) return false;
    if (object instanceof Boolean) return (boolean)object;
    return true;
  }

And as for the more complicated tests, you have to be able to
associate the value with
a JSON type anyway - number, string, object, array, true, false, null
- whether you're
dealing with a JsonElement type, or a Dictionary or List or Decimal or
struct Foo or whatever
type. If you can make that association, you can perform those tests.

Daniel

On Wed, Sep 8, 2021 at 8:21 PM Greg Dennis <[email-masked]> wrote:
> On Thu, 9 Sep 2021 at 11:13 am, Daniel P
> <[email-masked]> wrote:

---

From: Glyn Normington <[email-masked]>
Date: Thu, 9 Sep 2021 04:18:19 +0100

> What are the places where we want to have those implicit conversions?

Good question. Expressions like @.foo and $.foo test for the presence of
the key foo in an object (and are false for non-objects), so that leaves
array accesses (e.g. @[0]) and values such as @, $, and JSON literals such
as numbers, strings, booleans, and null.

There is no consensus for @ (
https://cburgmer.github.io/json-path-comparison/results/filter_expression_with_current_object.html),
booleans (
https://cburgmer.github.io/json-path-comparison/results/filter_expression_with_value_true.html,
https://cburgmer.github.io/json-path-comparison/results/filter_expression_with_value_false.html),
or null (
https://cburgmer.github.io/json-path-comparison/results/filter_expression_with_value_null.html).
Interestingly, the comparison project doesn't yet conceive of implicit
conversions for other JSON literals, array accesses, or $.

I personally don't see a great deal of value in providing implicit
conversions, but maybe that's because of my preference for strongly typed
languages. I'd just as soon make these syntax errors. Does anyone *want*
implicit conversions?

(I don't think strong or weak typing in the implementation language has any
relevance to whether we support implicit conversions. If we define implicit
conversions in the spec, they can be supported equally well in both
strongly and weakly typed languages.)

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 9 Sep 2021 07:45:30 +0200

On 2021-09-09, at 05:18, Glyn Normington <[email-masked]> wrote:
> 
> Expressions like @.foo and $.foo test for the presence of the key foo in an object (and are false for non-objects),

These expressions really return nodelists.
We could define an implicit conversion of nodelists to a Boolean value (empty nodelist => false, all other nodelists => true).

> so that leaves array accesses (e.g. @[0]) and values such as @, $, and JSON literals such as numbers, strings, booleans, and null.

I don’t see a difference between $.foo and $[0].

Why would I write a literal in a random type to have that implicitly converted to Boolean?

So maybe we can focus on the implicit conversion needed for using nodelists in a Boolean context.

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Thu, 9 Sep 2021 06:32:02 +0000 (UTC)

I think you're right here, Carsten.  In light of this, were going to need conversions from node lists to various types in order to support things like $.foo == 2.
Here we're expecting $.foo to return a node list with a single value, and that value is 2.
If we accept that we can use any JSON value as a constant, then we could have things like $.foo == [1, 'two'].  This raises an interesting scenario in which we can take this to mean either a node list with a single value of an array, or a node list with two items, 1 and 'two'.  Both could be considered valid, I expect.  Do we need some sort of operator to distinguish these cases?  Maybe something like $.foo == <1, 'two'> to mean a node list (the latter case)?
In any case it looks like we need implicit conversions at some level.  It's not impossible, just a bit cumbersome in some languages.
Greg

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 9 Sep 2021 08:43:35 +0200

> Here we're expecting $.foo to return a node list with a single value, and that value is 2.

As you note below, that is dangerous.
(Of course, if that reflects current practice, we’ll need to make it work.)

> If we accept that we can use any JSON value as a constant, then we could have things like $.foo == [1, 'two'].  This raises an interesting scenario in which we can take this to mean either a node list with a single value of an array, or a node list with two items, 1 and 'two'.  

I don’t think we want to implicitly convert a nodelist to an array of the value of the nodes.
Is that something that current practice suggests?

> Both could be considered valid, I expect.  Do we need some sort of operator to distinguish these cases?  Maybe something like $.foo == <1, 'two'> to mean a node list (the latter case)?

That would be one syntax.
We also might want to have operators for cases like “contains”.
Again, guided by current practice.

> In any case it looks like we need implicit conversions at some level.  It's not impossible, just a bit cumbersome in some languages.

The conversion could be innate in the type “nodelist”, or it could be a property of operators like “==“.  I’d rather see operators that are very specific as to what they do, but maybe “==“ (and friends) are the places to make this conversion.  So you never compare a nodelist, you always invoke a conversion into JSON values.  A nodelist that doesn’t support that conversion “fails” the comparison expression.

Just thinking out loud; this needs to be driven by examples.

Grüße, Carsten

---

From: Stefan Gössner <[email-masked]>
Date: Thu, 9 Sep 2021 10:07:19 +0200

Truthy / falsy defined for JavaScript is described here.

https://masteringjs.io/tutorials/fundamentals/truthy

(Greg)
> If we accept that we can use any JSON value as a constant, then we 
> could have things like $.foo == [1, 'two'].  This raises an 
> interesting scenario in which we can take this to mean either a node 
> list with a single value of an array, or a node list with two items, 1 
> and 'two'.  Both could be considered valid, I expect.  Do we need some 
> sort of operator to distinguish these cases?  Maybe something like 
> $.foo == <1, 'two'> to mean a node list (the latter case)?
>

We voted for comparisons of scalar values only, i.e. nodelists 
containing a single node while taking its value.

So current draft reads (at least for now):

 > Comparisons are restricted to primitive values (such as number, 
string, |true|, |false|, |null|). Comparisons with complex values will 
fail, i.e. no selection occurs.

 > Containment tests work with arrays and objects.

We might want to extend containment tests to also work with nodelists.

---

From: Daniel P <[email-masked]>
Date: Thu, 9 Sep 2021 09:40:30 -0400

> These expressions really return nodelists.

Perhaps in the WG's specification. But specifically regarding @.foo,
Goessner JavaScript does not return "nodelists", nor do any of the
implementations that use JavaScript, PHP or Python as a scripting
language. But also neither does the most widely used of the
implementations that specify expressions, Jayway JsonPath, which is
rigged into enterprise Java API's. Together, just these would account
for the majority of implementations in actual use, I would suggest the
vast majority.

For the JavaScript implementations, @ corresponds to a Javascript
variable, and @.foo corresponds to a key lookup in a Javascript
object, if that fails, it returns undefined, otherwise, it returns a
single value. Analogous observations can be made about other
implementations using dynamically typed languages for expression
evaluation.

For Jayway JsonPath, @.foo is treated as a JSONPath expression and is
evaluated to a single value following these rules:

- JsonPath selectors that can return multiple results (recursive
descent, unions, slices, wildcards, paths with expressions) always
return a JSON array, which can contain zero, one or many values.
- JsonPath selectors that can only return a single value (identifiers,
indexes), return a single value.

The result is a JSONPath expression evaluated to a single value. This
is obviously not equivalent to returning a "nodelist" and selecting
the 0th element (if it exists).

These observations suggest that it might be helpful to include a note
in the draft that clarifies the purpose of the WG. Is it to
standardize existing behaviour? What would that mean when the extant
implementations exhibit different behaviour, and because they are
entrenched in legacy API's, they cannot change? Or is it to invent
something new, inspired by the original, picking and choosing from
popular implementations as the WG sees fit? If something new, should
it have a different name?

Best regards,
Daniel

---

From: Glyn Normington <[email-masked]>
Date: Thu, 9 Sep 2021 19:37:44 +0100

On Thu, 9 Sept 2021 at 14:40, Daniel P <[email-masked]> wrote:
> standardize existing behaviour?

Yes. Please see our charter:  https://datatracker.ietf.org/wg/jsonpath/about/

> What would that mean when the extant
> entrenched in legacy API's, they cannot change?

The charter says:

"Where there are
differences, the working group will analyze those differences and
make choices that rough consensus considers technically best, with
an aim toward minimizing disruption among the different JSONPath
implementations."

So we don't have to preserve all existing behaviour. It will be up to
each implementation whether or not to migrate to the eventual
standard.

> Or is it to invent

I doubt we want to reinvent JMESPath.

---

From: Greg Dennis <[email-masked]>
Date: Fri, 10 Sep 2021 05:47:05 +0000 (UTC)

I've opened issue 119 (https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-base/issues/119) to continue this discussion.  I think it's easier to read along in an issue than trying to browse an email chain.
Greg
    On Friday, September 10, 2021, 06:38:07 AM GMT+12, Glyn Normington <[email-masked]> wrote:  

The charter says:

"Where there are

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 10 Sep 2021 08:43:28 +0200

On 9. Sep 2021, at 15:40, Daniel P <[email-masked]> wrote:
> 
> standardize existing behaviour

Of course!

But there is a difference between behavior and the model used to describe that behavior.
The nodelist is an abstraction that can be used to properly describe that existing behavior (or also to clean it up, if that is desired).

Of course we can make a difference between, say, $..foo being used as a selector in a JSONpath query expression itself and using the same in the expression language, say, in a filter expression or an index expression.

> object, if that fails, it returns undefined, otherwise,

The concept of “undefined” is foreign to JSON.
I think it is almost inevitable that, whatever we do, expression constructs in the JSONPath expression language will have ranges and domains that include values  like that (i.e., that are not JSON values).

Grüße, Carsten

---

From: Stefan Gössner <[email-masked]>
Date: Fri, 10 Sep 2021 12:38:05 +0200

The latest draft is quite clear:

 > The query is a sequence of zero or more /selectors/, each of which is 
applied to the result of the previous selector and provides input to the 
next selector. These results and inputs take the form of a /nodelist/, 
i.e., a sequence of zero or more nodes.

In my mental model a 'nodelist' is simply an abstract form of a sequence 
of nodes of the tree representation of the query's JSON argument. That 
abstract form is used by implementations internally and mostly coded via 
arrays then - which is implementation specific though.

I have a very strong opinion about fundamental working of JSONPath by 
"JSON in - JSON out". Only that way we can apply a JSONPath query to the 
result of another query ... implying a result array of course. 
Optionally the result array does not contain values, but normalized 
pathes of the selected nodes, which is a concrete instance of the 
abstract resulting nodelist then.

I see your post as a hint for us to perhaps concretise the 'nodelist' 
definition in the spec and check its use throughout the text.
--
sg