Subject: [Jsonpath] Some Comments ...

From: Stefan Gössner <[email-masked]>
Date: Fri, 26 Feb 2021 19:30:30 +0100

Hello List,

It has been important to go through this list threads carefully. In fact 
I should have done that at first. Now I can understand the current draft 
and appreciate the work already done much better.

I collected some citations (important from my point of view) with 
comments already in Markdown.

## Title of the specification

 > JSONPath: A query language for JSON data.
*(Carsten Bormann)*

 > I think I’d slightly prefer the term “syntax” to “language” because 
“query language” has a smell of various things that end with the letters 
“Q” and “L”.  But not passionate about that.
*(Tim Bray)*

 > JSONPath: A query syntax for JSON.
 > Another wild-card idea: JSONPath: Query expressions for jSON
*(Tim Bray)*

 > The beauty of this is that the plural form “query expressions” 
implies a set of expressions, so it implies “language”.  It’s indeed 
more than the grammar/syntax of those, so why not talk about the 
expressions as a whole.  This also makes it possible to just use “for 
JSON”, without going into detail what these query expressions operate on.
*(Carsten Bormann)*

There seems to be an agreement for "*JSONPath: Query expressions for 
JSON*". I like that also.

## Terminology

 > My own view is that the terminology should stay consistent with RFC
8259, and that the word "object" should not be used for items that are
not JSON objects in the sense of RFC 8259.
*(Daniel P)*

 > To Carsten's point about what we call things, the number of distinguished
terms per RFC8259 is pretty small: JSON text, value, object, array, number,
string.  Having spent quite a bit of time specifying JSON DSLs, I find that
using just those terms doesn't seem to get in the way or cause problems, so
I'd argue that we should stick to them (and build up to higher-level
constructs as required for JSONPath).
 >
 > … oh, and I forgot the very useful "member".
*(Tim Bray)*

 > … and “element” (the things in arrays). *(Carsten Bormann)*

 > The problem with JSON value is that it also can be quite confusing 
due to the usual use of that term.  Pointing to a tree and saying “the 
values inside that tree” is not going to be felt as equivalent to “the 
set of all subtrees of that tree, including the tree itself”.  But if 
JSON value is the only term we have, it has to be.  Hence my preference 
to talk about data items when I mean the items themselves and not their 
“value”.
*(Carsten Bormann)*

 > I think the key difficulty is whether each (key, value) pair in an 
object is "a thing" that can be identified and manipulated and 
potentially returned. (If we're talking analogies, then it's analogous 
to an attribute node in the XDM model).
*(Michael Kay)*

 > ECMA-404 uses "name/value pair", which is what I understand the term
"member" to mean (Douglas Crockford uses "member").
*(Daniel P)*

 > I think the term “union” is poor. If we think of it as concatenation 
of results, then the result is as expected.
*(Glyn Normington)*

I understand, that within RFC8259 we have JSON values of different 
types. They are structured somehow, which is not so much of interest here.

But while querying that structure with JSONPath it is vitally important 
to identify that hierarchical structure as a tree. So in fact we build 
up a higher-level construct here. We also need to call "the things" in 
the tree somehow. I was able to identify

* "node" or "item" of a tree
* "member" of an object
* "name/value" or "key/value" pair alias "member"
* "element" of an array

but could not see an agreement here.

I agree to Glyn calling the term "union" poor (s. below).

## Differentiation from JSON Pointer (JSONPath draft charter)

 > I anticipate being asked "Why is JSON Pointer not sufficient?" Indeed 
its abstract says:
 >
 >   JSON Pointer defines a string syntax for identifying a specific 
value within a JavaScript Object Notation (JSON) document.
 >
 >... which sounds awfully similar.  If we could include a sentence about
that, or a link to an answer, that might be helpful.
*(Murray S. Kucherawy)*

 > No - it's not similar in concept, they're separate things. If you 
really wanted to mention JSON Pointer, you could say something like 
"Note that while JSON Pointer (RFC xxxx) is already standardised, it is 
designed to provide a reference to a single, specific part of a JSON 
document, whereas JSONPath provides the ability to query a document and 
potentially return multiple values."
*(Mark Nottingham)*

 >The short answer is that JSON pointer is good if you already know the 
structure of the JSON data item you want to point into, and you want to 
point to exactly one position in there.  If you need to do something 
that is closer to a “search” (which might also result in multiple 
positions), JSONPath gives you more rope.
*(Carsten Bormann)*

+1

## References to XPath

 > I wonder if the analogies between XPath and JSONPath are going to be 
helpful, or whether they're actually dangerous by implying equivalences 
between constructs that are in fact somewhat different?
*(Michael Kay)*

 > I tend to agree. Although JSONPath was inspired by XPath, I wouldn't
want to confuse the JSONPath spec by going into detailed comparisons at
the risk of contradicting the normative text.
*(Glyn Normington)*

 > Someone on StackOverflow today asked a question about JSONPath; they 
called it (and tagged it) XPath, we really don't want that kind of 
confusion.
 >
 > In addition, the reference to the XPath specification in 6.2 is out 
of date, and the comparison with XPath in Table 2 is very approximate 
and the terminology inaccurate: for example there is a mention of "node 
sets", which exist in XPath 1.0 but not in XPath 2.0, yet the citation 
is to XPath 2.0. For someone who knows the semantics of XPath the 
comparison raises all sorts of questions about sorting of results into 
document order, elimination of duplicates etc, which are complications 
this spec can well do without. (Though some answers are needed, for 
example if ..store..price matches the same price in more than one way, 
do you get more than one result? And if not, what does "the same price" 
actually mean?)
*(Michael Kay)*

It seemed to be important in 2007, while argumenting to have something 
like XPath for JSON. If nowadays the terminology used has changed 
significantly with XPath 2.0 and 3.0, we better leave that comparison 
table 2 out. I am quite passionless here.

## Array Slice Operator

 > Thanks! The ABNF for an array slice in that reference
 > ```
 > integer = [%x2D] (%x30 / (%x31-39 *%x30-39))
 >
 > array-slice = [ integer ] ws %x3A ws [ integer ]
 >                     [ ws %x3A ws [ integer ] ]
 >                              ; start:end or start:end:step
 > ```
 > is consistent with JMESPath, Python, and my understanding of
ECMASCRIPT 4.
 > *(Daniel P)*

 > Did anyone else have an opinion on the behaviour of slices such as [::0]?
The current draft allows this and says it returns an empty array, but there
is good reason to say it should error so that the slice operation is then
consistent with Python slicing. See below for more context.
*(Glyn Normington)*

It's good having read this thread and thus understand the current draft 
much better. I like the decision to be consistent with Python and also 
getting an empty selection set with `step=0`.

FYI: there is a recent proposal for adding slice notation syntax to 
JavaScript, currently at stage 1 of the TC39 process.

https://github.com/tc39/proposal-slice-notation

Interestingly it won't have a step argument ...

https://github.com/tc39/proposal-slice-notation#why-doesnt-this-include-a-step-argument-like-python-does

... because of syntax collision with the new `this-binding` syntax 
proposal `::`

https://github.com/tc39/proposal-bind-operator

However, we should not let us influence by this.

## Unions

 > I don't think any implementation would remove duplicates from a path
such as `"$.store.book"`. I believe this is only somewhat controversial
in the context of unions [,]. The name "union" suggests that distinct
values be returned, compare with SQL unions. But Stefan Goessner's
implementation doesn't do that, it concatenates all results that meet
each criteria. There are a few JSONPath implementations that produce
real unions with no duplicates instead of concatenated results, but I
don't think that's the consensus.
*(Daniel P)*

 > I think the term “union” is poor. If we think of it as concatenation 
of results, then the result is as expected.
*(Glyn Normington)*

 > I agree with that comment, but it's partly because I'm used to SQL UNION,
which is different. I prefer the JMESPath term for an analogous construct,
MultiSelect List, https://jmespath.org/specification.html#multiselect-list.
*(Daniel P)*

Introducing the union operator `[,]` simply was meant an analogon to 
XPath's operator `'|'`. I cannot tell, if it was a simple combination of 
node sets in Xpath 1.0 or a true union without duplicates. I obviously 
was not aware of that subtle (essential ?) union characteristic.

So I fully agree to Glyn Normington's '... the term “union” is poor' 
statement. Are there some better alternative terms, perhaps 'multi-index 
operator', 'index list', 'subscript list', etc.?

## Duplicates and Ordering

 > It was my impression that we were talking about duplicated nodes not
duplicated values:
 >
 > Given th array [10,20,30]
 >
 > $..[0,1,0]
 >
 > Would yield only two results [10, 20]
 >
 > (Not that I'm advocating for removing duplicates, personally I think we
shouldn't)
*(Marko Mikulicic)*

 > You’re framing this as “removing duplicates”.
Another view is that [10, 20, 10] would be “adding duplicates” (copies 
of the same node). Related are ordering issues:
 >
 > `$..[1,0] ➔ [20, 10] Or [10, 20]`
 >
 > I would expect the spec will leaves implementations some leeway here, 
but that should be based on an examination of existing implementations.
*(Carsten Bormann)*

 > The mental model that leads to omitting duplicate nodes in the output is
"selection": if you take an input array and select nodes with index 0,1 or
0, you get only 2 results (since selecting an index twice has no effect).
 >
 >OTOH, if you opt for a "collect" model, whenever you encounter a node that
matches that query you add it to the result stream, thus the same nodes can
be present multiple times in the result.
 >
 >I have a slight preference for the "collect" model, because the general
case in jsonpath is to collect things that appear at various points in the
json tree. For example:
 >
 >`{"a": {"b": 1, "c": 2}, "d": 3},  $.a.b yields [1] and not 
{"a":{"b":1}}`
 >
 >(i.e. jsonpath is not a filter and view operation but a pick and gather
operation)
*(Marko Mikulicic)*

 > In implementations that support paths (the majority don't), the query
function takes a parameter that indicates values or paths. In both
cases the query returns a JSON array of JSON values, in the latter
case, a JSON array of normalized paths.
*(Daniel P)*

I must confess to never having thought about duplicates, let alone 
wanting to eliminate them. So I do like Marko's comparison of 
'selection-model' vs. 'collection-model' a lot. I would opt for the 
latter. In this sense the result of a 'JSONPath query expression' should 
be termed a 'collection'.

Regarding ordering I see something like a 'natural ordering', according 
to which

`$..[0,1] ➔ [10, 20]`
`$..[1,0] ➔ [20, 10]`

would result with the example above.

I do understand the use cases for reordering, duplicates removal, 
filtering, etc.. But this can always be seen as a postprocessing step on 
the resulting collection by handing it over to accompanying tools (think 
of pipe operator).

Of course this cannot work on the result collection of values alone (s. 
duplicate nodes vs. duplicate values above), it rather requires a 
collection of (normalized) pathes. In this sense, I like this view:

 > In my opinion the right balance between powerfulness and enabling
simple implementations has been so far one of the key factors that
made JSONPath popular over other alternatives, even if it lacks
support for aggregation functions.
*(Davide Bettio)*

## Filter Expressions

 > Related to that, it would be helpful to determine if JSONPath filters
apply to both JSON objects and arrays, or only to JSON arrays.
*(Daniel P)*

 > I would support restricting filters to arrays, if others agree.
*(Glyn Normington)*

I tend to let implementations and their "normative force of the factual" 
decide here or in doubt agree to Glyn's restriction to arrays.

I am very unhappy with confusing `$..book[(@.length-1)]`, where `'@'` 
addresses the array itself and implies that array has a `length` 
property. In filter expression examples `'@'` more consistently 
addresses the current array element.

The invocation of 'the underlying scripting engine' wasn't meant a 
serious normative aspect, but rather a quick and dirty solution for 
JavaScript and PHP implementations at that time.

### Corner Case

 > Consider this perfectly legal JSON object
 >
 > ```{ "ab": 0,  "'a.b": 1,  "a-b": 2, "a": { "b": 3 } }```
 >
 >So `$.ab` is 0, `$.a.b` is 3, `$['a.b']` is 1, `$['a-b']` is 2. You'd 
like to say `$.a-b` but lots of libraries will refuse it because `"a-b"` 
is not a legal JavaScript "name" construct, that's why you have to say 
`$['a-b']`.
 >
 > But suppose your library would accept `$.a-b`.  Then `$.a-b` and 
`$['a-b']` would be synonyms, but `$.a.b` and `$['a.b']` wouldn't.
*(Tim Bray)*

Hmm ... this seems to be a hint to better exclude `'-'` from 
dot-child-selector syntax. I think I have read more discussion about 
that, currently don't know where.

## Respect Implementations

 > As I mentioned in the session, I think there's a non-trivial amount 
of risk here that some implementations won't be willing or able to move 
away from their current behaviours, even if interoperability would 
improve if they did so. However, there are ways to mitigate that (e.g., 
a separate 'rfcxxxx compliant' mode). Even so, it will be important to 
get good participation from as many current implementers as possible.
*(Mark Nottingham)*

 > The WG will develop a standards-track JSONPath specification that
is technically sound and complete, based on the common semantics
and other aspects of existing implementations.  Where there are
differences, the working group will analyze those differences and
make choices that rough consensus considers technically best, with
an aim toward minimizing disruption among the different JSONPath
implementations.
*(Barry Leiba)*

 > I'm OK with this, but for context: I've been a pretty intense 
JSONPath user
in recent years, and AFAIK the spec, and the implementations, are mostly
OK, so the choice between "make JSONPath good" and "don't invalidate
implementations" is unlikely to come up. If it did, my predisposition would
be to err on the side of not breaking implementations, but I don't think
that's inconsistent with Barry's text.
*(Tim Bray)*

+1 to all.

## Error Handling

 > My mental model at the moment is that a JSONPath expression can be 
valid or erroneous; application of a valid expression yields a result 
(which may be empty), but does not raise errors.  That may not be the 
right model for all applications.
*(Carsten Bormann)*

 > The  general approach that I've seen several times (including my
Elixir implementation) is that an error is raised when there is a
syntax error, therefore an invalid expression (e.g. $.foo[[5]) raises
an error. Conversely a valid expression applied to a bogus input never
raises an error (e.g. `$.foo.bar on "test" evals as []`).
*(Davide Bettio)*

 > On the whole I think JSONPath is designed to be "forgiving", i.e. 
such things aren't errors, e.g. I think I read in the spec that 
filtering a non-array isn't an error, it's some kind of no-op. That 
approach isn't always best for everyone, but it's important to be 
consistent.
*(Michael Kay)*

 > I would expect one component of this policy to be:
 >
 > Whether a JSONPath query is valid or not does not depend on the 
arguments it is applied to.
 >
 > I.e., you can look at the query and find out independently, without 
knowing any data, whether it is valid or not.
*(Carsten Bormann)*

I like and totally agree with the *forgiving mental model*, so having  
only syntax errors, which do not dependent on data.

Thanks
--
sg

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 19:12:12 +0000 (UTC)

Hello List!  I appears I'm finally getting these! Woohoo!
Stefan, thank you for your input.  However it seems that the MD formatting (as much as I appreciate the effort) has been lost and I'm having a hard time parsing quoted text vs your responses in the text as it appears the email server completely removed any formatting and then further reformatted the raw text to ensure that it is compatible with email clients from 25 years ago.
I feel that email is probably the least ideal format for conversations like this.  The thing is, we have a wonderful GitHub repository (https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath) complete with issues and PRs, mechanisms that are specifically DESIGNED for this sort of discourse.
We also have a Slack workspace (https://join.slack.com/t/jsonpath-standard/shared_invite/zt-mxse9mgl-DK~35JawDjcsFUY9PVO5~Q) which is great for small side conversations and general questions from the public.  (Though it appears that the workspace has been removed from the README.)  The nice thing about Slack is the integrations.  It's currrently set up to monitor the GitHub repo for new issues and PRs and StackOverflow for new "json-path"-labeled questions.
Both of these support formatting (with their own flavors of Markdown).
I'm not saying that we should abandon the emails list.  But it's not great for this kind of discussion.
It's also worth saying that I've been part of the JSON Schema specification development for over five years, and using both GitHub and Slack has been invaluable.
Thanks for the effort put into this so far by everyone.
Greg
    On Saturday, February 27, 2021, 07:30:41 AM GMT+13, Stefan Gössner <[email-masked]> wrote:  
 
 Hello List,

*(Carsten Bormann)*

*(Tim Bray)*

## Terminology

*(Tim Bray)*

+1

## References to XPath

## Array Slice Operator

ECMASCRIPT 4.

## Unions

shouldn't)
*(Marko Mikulicic)*

{"a":{"b":1}}`
operation)
*(Marko Mikulicic)*

`$..[0,1] ➔ [10, 20]`
`$..[1,0] ➔ [20, 10]`

## Filter Expressions

*(Glyn Normington)*

### Corner Case

JSONPath user

+1 to all.

## Error Handling

Thanks
--
sg

-- 
Jsonpath mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/jsonpath

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 20:34:30 +0100

> We also have a Slack workspace (https://join.slack.com/t/jsonpath-standard/shared_invite/zt-mxse9mgl-DK~35JawDjcsFUY9PVO5~Q) which is great for small side conversations and general questions from t…

Do we have a way to get these conversations archived?

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 19:52:03 +0000 (UTC)

If you're referring to the 10,000 message limit for the free offering, that's a valid point against Slack.  But it doesn't provide merit toward email nor detract from GitHub, which does "archive" discussions.
Greg

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 20:55:47 +0100

> If you're referring to the 10,000 message limit for the free offering, that's a valid point against Slack.  But it doesn't provide merit toward email nor detract from GitHub, which does "archive" d…

I wasn’t arguing anything; I just want to make sure that if we adopt slack the slack discussions are not lost when slack goes belly-up.  Email is handled by the IETF itself, for github we have some schemes for making backups of non-git information (git itself of course is a giant backup system).  For jabber, which (for better or worse) is still the supported IM system, we again have backups at the IETF.

So what’s the backup story for slack?

Grüße, Carsten

> 
> Greg
> 
> On Sat, 27 Feb 2021 at 8:34 am, Carsten Bormann
> <[email-masked]> wrote:

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 19:58:31 +0000 (UTC)

It's my understanding that Slack backs up everything so that WHEN you decide to pay for it, you get all of your "lost" history back.
Greg 

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 21:06:48 +0100

> It's my understanding that Slack backs up everything so that WHEN you decide to pay for it, you get all of your "lost" history back.

That is not answering my question.  We need to have backups that are independent of the vicissitudes of corporate odysseys.  

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 20:14:36 +0000 (UTC)

That's likely the extent of backups for Slack.  I'm guessing that you're dissatisfied with this, which means that we're down to GitHub.
However, I would argue also that not all conversations NEED to be archived.  General questions, like those which appear on StackOverflow, occur in the Schema Slack ALL THE TIME, but these don't need to be archived.  We also use it as a space to work out details of new features before publicizing them.  It has its place.
Greg

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 21:28:04 +0100

> That's likely the extent of backups for Slack.  I'm guessing that you're dissatisfied with this, which means that we're down to GitHub.

We built something for (the non-git parts of) github; we might build something for slack.  I’m just trying to find out if there already is anything we can use.
(We created a whole WG for understanding how to use github in the IETF; we might want to do the same for slack at some point.)

> However, I would argue also that not all conversations NEED to be archived.  General questions, like those which appear on StackOverflow, occur in the Schema Slack ALL THE TIME, but these don't nee…

I don’t think that works.  Before long, someone will claim that issue X was discussed in slack and doesn’t need to be rediscussed in the archived media.
Or, worse, some key people will have followed that conversation, agree on the archival media, and the whole background is lost when (not: if) slack experiences the next corporate upset.

Writing specifications in an IETF context leads to a surprising need for longevity.
I only recently spent a few hours digging out some detail about RFC 3095 that we nailed down around September 2000.  Fortunately, someone had pulled a copy of the archive on a mail server we used at the time (most IETF mail has since migrated to the IETF).
Being able to trace back discussions to an archived context is core to the value of building standards for decades.

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 21:42:12 +0000 (UTC)

> I don’t think that works. Before long, someone will claim that issue X was discussed in slack and doesn’t need to be rediscussed in the archived media.

It hasn't been much of an issue for us in JSON Schema.  There have been times when I'd have liked to find older conversations, but it's rare.
But also, if I email you directly, it doesn't appear in the archive.  We could have entire conversations and discuss decisions that only appear in an email chain between us.
Furthermore, if I just need clarification on a topic, I don't need (or necessarily want) that publicized.  It's much better in a medium like Slack.  (If that conversation leads to a change in the spec, then that change should be discussed publicly, possibly quoting a non-archived conversation as necessary.)
Lastly, in my experience conversations can become quite heated.  This is stuff you really don't want archived as much of it is unhelpful.
The difference is that we want discussions that can influence the spec archived.  Anything else is superfluous.  If we make a policy that such conversations MUST be summarized (at least) in an archival medium (here or GitHub), then both of our conditions are satisfied: you get archives of the important stuff, and we also get an easier comms mechanism.
Greg
    On Saturday, February 27, 2021, 09:28:12 AM GMT+13, Carsten Bormann <[email-masked]> wrote:  

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 22:59:11 +0100

Hi Greg,

at this point I’m going to stop supplying additional input to this discussion.

I think I made my point why it is useful to automatically, effortlessly archive the WG proceedings.
We have a lot of manual “policies” of the kind you are suggesting.
Of course, they are not working.  Why should they?

I did not provide an opinion on using slack (just in case you care: I personally don’t like that specific tool, but the WG as a whole has to choose mechanisms it is comfortable with, and I don’t want to rain on the party of the people who do like it).  This is for the chairs to decide.

I’d still need an answer for my question, which is whether it is possible to maintain independent archive copies of slack discussions.  Maybe we can find out from other IETF groups that are contemplating using slack.

Grüße, Carsten

> On 2021-02-26, at 22:42, Greg Dennis <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 23:23:29 +0100

Hi Stefan,

> On 2021-02-26, at 19:30, Stefan Gössner <[email-masked]> wrote:
> 
> Hello List,
> 
> It has been important to go through this list threads carefully.

Great work, thank you!

(And thank you for doing it in markdown, which made this collection of various inputs quite readable.)

I think there are a few cases where we have strong consensus and just need to update the document (issues will come up when that is done, because writing things up brings issues to the forefront, but it doesn’t make sense to discuss them before draft text is written down).

The issue of “duplicates”, collection/selection etc. really is one of defining the processing model.  (A *model*, so not unduly restricting implementations, but guiding a common understanding of what is going on nonetheless.)  I’m still troubled imagining a processing model that has all the properties that have been brought up.  So maybe we can focus on this for a while, maybe based on strawman models that people bring up — but not just by simply listing desirable properties without knowing how they fit together.

My 2 ¢ for the moment.  
I know that I need to ramp up my time budget for this group again…

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 22:42:32 +0000 (UTC)

It's the copy in the primary branch in GitHub the current state of the spec?  If not, where can it be found?
Greg

  On Sat, 27 Feb 2021 at 11:23 am, Carsten Bormann<[email-masked]> wrote:   Hi Stefan,

Great work, thank you!

My 2 ¢ for the moment.  

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 26 Feb 2021 23:52:39 +0100

> It's the copy in the primary branch in GitHub the current state of the spec?  If not, where can it be found?

There are various levels of draftiness.

In increasing order:

(0)
RFC (which we don’t have yet :-)

(1)
https://datatracker.ietf.org/doc/draft-normington-jsonpath/
(The “Internet-Draft”, i.e., the official WG working document.)

(2)
Main branch (previously caller master branch) of WG repo,
https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath
(The markdown document is the source, the other files are generated versions.
This hasn’t been converted to Martin Thomson’s template, which keeps generated versions in a separate branch so the main branch can be clean.)

(3)
The outstanding fixes in PR #46
https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath/pull/46

I could continue hacking that PR, e.g, based on Stefan’s summary, or the authors could merge it and then we could have separate PRs to implement the items listed in Stefan’s summary.  (I would prefer the latter, as it is less work for everyone.)

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Fri, 26 Feb 2021 23:36:16 +0000 (UTC)

Thank you.  It should be noted that there are a lot of outstanding questions/comments on the PR (3) as well as it containing merge conflicts.
I'd like to have a poke around these documents and see if there's anything more to address that hasn't been already.
Greg
    On Saturday, February 27, 2021, 11:52:42 AM GMT+13, Carsten Bormann <[email-masked]> wrote:  

In increasing order:

(0)

(1)

(2)

(3)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 27 Feb 2021 02:05:03 +0100

> Thank you.  It should be noted that there are a lot of outstanding questions/comments on the PR (3) as well as it containing merge conflicts.

Fixed.  Can we merge?  I’d like to go through Stefan’s items next, as separate PRs if possible.

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Sat, 27 Feb 2021 01:21:11 +0000 (UTC)

I've approved the PR.
    On Saturday, February 27, 2021, 02:05:20 PM GMT+13, Carsten Bormann <[email-masked]> wrote:  

Grüße, Carsten

---

From: Stefan Gössner <[email-masked]>
Date: Sat, 27 Feb 2021 09:44:29 +0100

I copied these comments forward to Github ... for better reading and 
further commenting.

https://github.com/ietf-wg-jsonpath/draft-ietf-jsonpath-jsonpath/issues/54

I would also like to try out that cool Github discussion tab, Daniel 
wrote about.

thanks
--
sg

Am 26.02.2021 um 19:30 schrieb Stefan Gössner:
> Hello List,
>
> It has been important to go through this list threads carefully. In 
> fact I should have done that at first. Now I can understand the 
> current draft and appreciate the work already done much better.
>
> “query language” has a smell of various things that end with the 
> letters “Q” and “L”.  But not passionate about that.
> *(Tim Bray)*
>
> JSON value is the only term we have, it has to be.  Hence my 
> preference to talk about data items when I mean the items themselves 
> and not their “value”.
> *(Carsten Bormann)*
>
> But while querying that structure with JSONPath it is vitally 
> important to identify that hierarchical structure as a tree. So in 
> fact we build up a higher-level construct here. We also need to call 
> "the things" in the tree somehow. I was able to identify
>
> > I anticipate being asked "Why is JSON Pointer not sufficient?" 
> Indeed its abstract says:
> >
> "Note that while JSON Pointer (RFC xxxx) is already standardised, it 
> is designed to provide a reference to a single, specific part of a 
> JSON document, whereas JSONPath provides the ability to query a 
> document and potentially return multiple values."
> *(Mark Nottingham)*
>
> structure of the JSON data item you want to point into, and you want 
> to point to exactly one position in there.  If you need to do 
> something that is closer to a “search” (which might also result in 
> multiple positions), JSONPath gives you more rope.
> *(Carsten Bormann)*
>
> +1
>
> ## References to XPath
>
> helpful, or whether they're actually dangerous by implying 
> equivalences between constructs that are in fact somewhat different?
> *(Michael Kay)*
>
> and the terminology inaccurate: for example there is a mention of 
> "node sets", which exist in XPath 1.0 but not in XPath 2.0, yet the 
> citation is to XPath 2.0. For someone who knows the semantics of XPath 
> the comparison raises all sorts of questions about sorting of results 
> into document order, elimination of duplicates etc, which are 
> complications this spec can well do without. (Though some answers are 
> needed, for example if ..store..price matches the same price in more 
> than one way, do you get more than one result? And if not, what does 
> "the same price" actually mean?)
> *(Michael Kay)*
>
> It's good having read this thread and thus understand the current 
> draft much better. I like the decision to be consistent with Python 
> and also getting an empty selection set with `step=0`.
>
> MultiSelect List, 
> https://jmespath.org/specification.html#multiselect-list.
> *(Daniel P)*
>
> XPath's operator `'|'`. I cannot tell, if it was a simple combination 
> of node sets in Xpath 1.0 or a true union without duplicates. I 
> obviously was not aware of that subtle (essential ?) union 
> characteristic.
>
> statement. Are there some better alternative terms, perhaps 
> 'multi-index operator', 'index list', 'subscript list', etc.?
>
> > I would expect the spec will leaves implementations some leeway 
> here, but that should be based on an examination of existing 
> implementations.
> *(Carsten Bormann)*
>
> latter. In this sense the result of a 'JSONPath query expression' 
> should be termed a 'collection'.
>
> Regarding ordering I see something like a 'natural ordering', 
> according to which
>
> `$..[0,1] ➔ [10, 20]`
> `$..[1,0] ➔ [20, 10]`
>
> filtering, etc.. But this can always be seen as a postprocessing step 
> on the resulting collection by handing it over to accompanying tools 
> (think of pipe operator).
>
> Of course this cannot work on the result collection of values alone 
> (s. duplicate nodes vs. duplicate values above), it rather requires a 
> I tend to let implementations and their "normative force of the 
> factual" decide here or in doubt agree to Glyn's restriction to arrays.
>
> like to say `$.a-b` but lots of libraries will refuse it because 
> `"a-b"` is not a legal JavaScript "name" construct, that's why you 
> have to say `$['a-b']`.
> >
> of risk here that some implementations won't be willing or able to 
> move away from their current behaviours, even if interoperability 
> would improve if they did so. However, there are ways to mitigate that 
> (e.g., a separate 'rfcxxxx compliant' mode). Even so, it will be 
> important to get good participation from as many current implementers 
> as possible.
> *(Mark Nottingham)*
>

---

From: Tim Bray <[email-masked]>
Date: Sat, 27 Feb 2021 07:43:28 -0800

On behalf of James and mysell, co-chair hats on: First of all, thanks to
Stefan for pulling together that nice list of issues.  It needs to be
broken up into a bunch of separate issues we can argue about and resolve.
If either Stefan or Marko wants to do that fine, otherwise I will in the
next couple of days.

Secondly, on the subject of WG communication:

The WG mailing list isn't optional, there are those who will want to
participate and not deal with GitHub etc, they're within their rights.
It's archived and searchable and paid for by IETF.

The use of GitHub is something of an open question, but it has been
successfully applied in multiple WGs, including some of our most
highly-trafficked and contentious, to the extent that there are RFCs [8874
<https://tools.ietf.org/html/rfc8874>, 8875
<https://tools.ietf.org/html/rfc8875>] covering how to do it.  For the
purposes of WG work, James and I would like to combine the mailing list and
GitHub in the way that seems to have proven to work.  We can't forbid
anyone from having Slack conversations but that's entirely optional, you
don't have to go there to participate in the WG.

On Sat, Feb 27, 2021 at 12:44 AM Stefan Gössner <[email-masked]> wrote:

> I would also like to try out that cool Github discussion tab, Daniel wrote
> about.
>
> thanks
> --
> sg
>
>
> It has been important to go through this list threads carefully. In fact I
> should have done that at first. Now I can understand the current draft and
> appreciate the work already done much better.
>
> I collected some citations (important from my point of view) with comments
> already in Markdown.
>
>
> > The beauty of this is that the plural form “query expressions” implies a
> set of expressions, so it implies “language”.  It’s indeed more than the
> grammar/syntax of those, so why not talk about the expressions as a whole.
> This also makes it possible to just use “for JSON”, without going into
> detail what these query expressions operate on.
> *(Carsten Bormann)*
>
> > The problem with JSON value is that it also can be quite confusing due
> to the usual use of that term.  Pointing to a tree and saying “the values
> inside that tree” is not going to be felt as equivalent to “the set of all
> subtrees of that tree, including the tree itself”.  But if JSON value is
> the only term we have, it has to be.  Hence my preference to talk about
> data items when I mean the items themselves and not their “value”.
> *(Carsten Bormann)*
>
> object is "a thing" that can be identified and manipulated and potentially
> returned. (If we're talking analogies, then it's analogous to an attribute
> node in the XDM model).
> *(Michael Kay)*
>
> > I think the term “union” is poor. If we think of it as concatenation of
> results, then the result is as expected.
> *(Glyn Normington)*
>
> I understand, that within RFC8259 we have JSON values of different types.
> They are structured somehow, which is not so much of interest here.
>
> But while querying that structure with JSONPath it is vitally important to
> identify that hierarchical structure as a tree. So in fact we build up a
> higher-level construct here. We also need to call "the things" in the tree
> somehow. I was able to identify
>
> >   JSON Pointer defines a string syntax for identifying a specific value
> within a JavaScript Object Notation (JSON) document.
> >
> > No - it's not similar in concept, they're separate things. If you really
> wanted to mention JSON Pointer, you could say something like "Note that
> while JSON Pointer (RFC xxxx) is already standardised, it is designed to
> provide a reference to a single, specific part of a JSON document, whereas
> JSONPath provides the ability to query a document and potentially return
> multiple values."
> *(Mark Nottingham)*
>
> point to exactly one position in there.  If you need to do something that
> is closer to a “search” (which might also result in multiple positions),
> JSONPath gives you more rope.
> *(Carsten Bormann)*
>
> +1
>
> ## References to XPath
>
> > In addition, the reference to the XPath specification in 6.2 is out of
> date, and the comparison with XPath in Table 2 is very approximate and the
> terminology inaccurate: for example there is a mention of "node sets",
> which exist in XPath 1.0 but not in XPath 2.0, yet the citation is to XPath
> 2.0. For someone who knows the semantics of XPath the comparison raises all
> sorts of questions about sorting of results into document order,
> elimination of duplicates etc, which are complications this spec can well
> do without. (Though some answers are needed, for example if ..store..price
> matches the same price in more than one way, do you get more than one
> result? And if not, what does "the same price" actually mean?)
> *(Michael Kay)*
>
> significantly with XPath 2.0 and 3.0, we better leave that comparison table
> 2 out. I am quite passionless here.
>
> ## Array Slice Operator
>
> > I think the term “union” is poor. If we think of it as concatenation of
> results, then the result is as expected.
> *(Glyn Normington)*
>
> node sets in Xpath 1.0 or a true union without duplicates. I obviously was
> not aware of that subtle (essential ?) union characteristic.
>
> Another view is that [10, 20, 10] would be “adding duplicates” (copies of
> the same node). Related are ordering issues:
> >
> I must confess to never having thought about duplicates, let alone wanting
> to eliminate them. So I do like Marko's comparison of 'selection-model' vs.
> 'collection-model' a lot. I would opt for the latter. In this sense the
> result of a 'JSONPath query expression' should be termed a 'collection'.
>
> Regarding ordering I see something like a 'natural ordering', according to
> which
>
> `$..[0,1] ➔ [10, 20]`
> `$..[1,0] ➔ [20, 10]`
>
> the resulting collection by handing it over to accompanying tools (think of
> pipe operator).
>
> addresses the array itself and implies that array has a `length` property.
> In filter expression examples `'@'` more consistently addresses the current
> array element.
>
> The invocation of 'the underlying scripting engine' wasn't meant a serious
> normative aspect, but rather a quick and dirty solution for JavaScript and
> PHP implementations at that time.
>
>
> ### Corner Case
>
> like to say `$.a-b` but lots of libraries will refuse it because `"a-b"` is
> not a legal JavaScript "name" construct, that's why you have to say
> `$['a-b']`.
> >
> dot-child-selector syntax. I think I have read more discussion about that,
> currently don't know where.
>
> > As I mentioned in the session, I think there's a non-trivial amount of
> risk here that some implementations won't be willing or able to move away
> from their current behaviours, even if interoperability would improve if
> they did so. However, there are ways to mitigate that (e.g., a separate
> 'rfcxxxx compliant' mode). Even so, it will be important to get good
> participation from as many current implementers as possible.
> *(Mark Nottingham)*
>
> > I'm OK with this, but for context: I've been a pretty intense JSONPath
> user
> > My mental model at the moment is that a JSONPath expression can be valid
> or erroneous; application of a valid expression yields a result (which may
> be empty), but does not raise errors.  That may not be the right model for
> all applications.
> *(Carsten Bormann)*
>
> > On the whole I think JSONPath is designed to be "forgiving", i.e. such
> things aren't errors, e.g. I think I read in the spec that filtering a
> non-array isn't an error, it's some kind of no-op. That approach isn't
> always best for everyone, but it's important to be consistent.
> *(Michael Kay)*
>
> I like and totally agree with the **forgiving mental model**, so having

---

From: Glyn Normington <[email-masked]>
Date: Sat, 27 Feb 2021 17:18:48 +0000

I created the jsonpath-standard Slack workspace merely to get a 
conversation going about standardising JSONPath. I've toyed with 
deleting that workspace, but one or two people seem to find it useful, 
so I've held off.

However Slack is not great for several reasons including:

* it's ephemeral (since we can't pay for a license indefinitely and 
there is no "free" for open source, etc. pricing option)

* its contents don't show up in internet searches

* access to it depends on proprietary code which will probably be 
superseded by something else in a decade or two, if not sooner.

https://github.com/dutchcoders/slackarchive could be part of a solution 
to some of these problems, but it would need investment and would also 
need hosting somewhere, such as at the IETF.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 27 Feb 2021 20:14:25 +0100

On 2021-02-27, at 18:18, Glyn Normington <[email-masked]> wrote:
> 
> I've toyed with deleting that workspace, but one or two people seem to find it useful, so I've held off.

Fragmentation is a big problem in standardization groups.
It should not be stimulated lightly.

Grüße, Carsten

---

From: Greg Dennis <[email-masked]>
Date: Sat, 27 Feb 2021 20:37:26 +0000 (UTC)

All of those points against Slack are valid.  However, I have found that it has been invaluable in building a community around JSON Schema that otherwise wouldn't have existed.
- End users get to ask questions, even if just curious and want to explore the technology.- Implementation providers (like myself) get to share approaches, which results in stronger implementations.- Unrelated personal conversations occur that help people get to know each other (this DEFINITELY doesn't need to be archived).  This is often facilitated by private channels and conversations.
The point behind the Slack workspace is that it builds community.  Community is essential to the success of a project like this.  If you have issue with Slack specifically, whatever, but there needs to be SOME casual space for off-hand discussion.  Gitter or Discourse are also great options for this.  An email list will NOT create a community like these tools can.
I've built relationships with the maintainers that I never would have had without a more social discussion platform.  I've even earned their trust and become a co-author of the spec myself.
If you're worried about discussions that need to be archived occurring over these tools, we need to be open, explicit, and strict about pushing people to move to an archival medium.  It's not a difficult thing to do, and I'm happy to help enforce such a policy.  As I mentioned, we have been doing this for years at JSON Schema.  We're aware that conversations disappear after some time, and it's actually a driver for us to make sure important issues get out to GitHub.
Your welcome to come see for yourself.  https://join.slack.com/t/json-schema/shared_invite/zt-8o78yech-XtsW6g9hrqe0SVq94Jim8Q
Greg 

Grüße, Carsten