Subject: [Json] json-text-sequence-06

From: Nico Williams <[email-masked]>
Date: Fri, 22 Aug 2014 01:32:17 -0500

URL:            http://www.ietf.org/internet-drafts/draft-ietf-json-text-sequence-06.txt
Status:         https://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/
Htmlized:       http://tools.ietf.org/html/draft-ietf-json-text-sequence-06
Diff:           http://www.ietf.org/rfcdiff?url2=draft-ietf-json-text-sequence-06

Changes:

 - removed section 1.1
 - defined RS
 - made RS a MUST (there was no sense in it being a SHOULD, right?)
 - added a note about the utility of (with no recommendation to)
   emitting an LF at the end of each JSON text
 - added an acknowledgement of Carsten's idea of using a non-ws ASCII
   control character as the separator.

The only changes I can think of wanting to make are:

 - add a reference for RS

 - tweak the ABNF: *(1*RS JSON-text) or *(RS JSON-text) or something
   like *(RS (JSON-text / invalid-JSON-text)) (without defining
   invalid-JSON-text) ?

Nico
--

---

From: Dominik Tomaszuk <[email-masked]>
Date: Fri, 22 Aug 2014 19:12:49 +0200

>     like *(RS (JSON-text / invalid-JSON-text)) (without defining

I propose to avoid all delimiters from C0. This would allow the use of 
the hierarchical levels, where US is the lowest level and FS is the 
highest level.

My proposal in ABNF:

JSON-sequence = *(sep JSON-text)
sep = *(
   %x1C /  ; File Separator
   %x1D /  ; Group Separator
   %x1E /  ; Record Separator
   %x1F )  ; Unit Separator

Dominik Tomaszuk
Research Fellow
University of Bialystok
Poland

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 22 Aug 2014 10:34:14 -0700

Greetings again. The first WG Last Call ended up with no consensus about whether the document was useful as it stood. Nico has made a few rounds of changes, and so I'd like to ask folks to read it again, fresh.

Please respond on the list whether the technical proposal in the new document could be useful in meeting the use cases and scenarios described in the document. Note that this is *not* a request for comments about whether or not you liked the old proposal better; it is a request about the current document.

The current document:

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Fri, 22 Aug 2014 15:59:19 -0500

On Fri, Aug 22, 2014 at 07:12:49PM +0200, Dominik Tomaszuk wrote:
> I propose to avoid all delimiters from C0. This would allow the use
> of the hierarchical levels, where US is the lowest level and FS is
> the highest level.

Did you mean "use" instead of "avoid"?  Otherwise you seem to contradict
yourself.

>   %x1E /  ; Record Separator

That could be an interesting MIME type, but not this one.

This format is just a sequence, a very _flat_ sequence.  Use cases
include logfiles (flat), indeterminate-length database query results
(also flat), and really, any case where the alternative would be to
return a long JSON array at the top-level.

A format that sequences arrays at all levels wouldn't be easy to
implement.  It'd be easier to implement an incremental, streaming JSON
parser instead.

Whereas flat sequences can be parsed with any off-the-shelf JSON
parsers, and you still get a very usable degree of incremental and
streaming parsing without having to go the distance at the JSON parser
level.

Nico
--

---

From: Dominik Tomaszuk <[email-masked]>
Date: Sat, 23 Aug 2014 10:58:12 +0200

> Did you mean "use" instead of "avoid"?  Otherwise you seem to contradict
Right. I made a mistake, a slip of the tongue of course. Instead of 
"avoid" should be "allow".

> return a long JSON array at the top-level.
Agree. But this flat sequence may have groups, i.e.
A [RS] B [US] C [RS] D
Here you can process to the first delimiter [RS], or move on and process 
to the first delimiter [US] as a group A [RS] B.
You decide whether you take only A or A and B. (This may depend on 
various factors, i.e. how large is logfile.)

> implement.  It'd be easier to implement an incremental, streaming JSON
+1

> streaming parsing without having to go the distance at the JSON parser
You can do it also with this group separators.

Dominik Tomaszuk
Research Fellow
University of Bialystok
Poland

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 23 Aug 2014 08:08:16 -0700

As much as it is fun to keep debating old topics, it would help the WG if folks could focus on the current work. We would like to finish up, and the following WG LC is a way for us to determine how close we are  to that.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Sat, 23 Aug 2014 10:57:02 -0700

Comments on http://tools.ietf.org/html/draft-ietf-json-text-sequence-06

Section 2.

   Two or more
   RS characters in sequence do not denote "empty" nor missing JSON
   texts.

Do we need to say this? Why?

==========================

   JSON text sequence encoders MAY emit an RS after emitting a
   JSON text.

What is this for?  Was this in previous versions?  Why would an implementer
do this?  I can’t perceive any benefits.

==========================

2.2

   In some contexts it is useful to write an LF (%x1A) after writing a
   JSON text: it makes working with line-oriented text tools easier.

First of all, that’s %x0A not 1A.

I actually don’t believe this statement is true, but perhaps I’m in a
minority.  In the general case, you just can’t process JSON with grep and
expect to get a useful result.  Sometimes you will in the special case
where you know the peculiarities of the JSON some particular software is
generating.  But if it were my shop, I’d worry that someone would achieve a
useful result with grep, put it into a production job, and there’d be a
catastrophic failure 18 months later when an auto-update to the
JSON-generating library starts introducing newlines between member names
and values.

The *right* way to accomplish this goal is to pull out the JSON texts
between the separators and run them through a JSON prettyprinter that
produces a deterministic output, then go crazy with grep and friends.

On Sat, Aug 23, 2014 at 8:08 AM, Paul Hoffman <[email-masked]> wrote:

> As much as it is fun to keep debating old topics, it would help the WG if
> folks could focus on the current work. We would like to finish up, and the
> following WG LC is a way for us to determine how close we are  to that.
>
> --Paul Hoffman
>
> > Greetings again. The first WG Last Call ended up with no consensus about
> whether the document was useful as it stood. Nico has made a few rounds of
> changes, and so I'd like to ask folks to read it again, fresh.
> >
> > Please respond on the list whether the technical proposal in the new
> document could be useful in meeting the use cases and scenarios described
> in the document. Note that this is *not* a request for comments about
> whether or not you liked the old proposal better; it is a request about the
> current document.
> >
> > The current document:
> >
> > URL:
> http://www.ietf.org/internet-drafts/draft-ietf-json-text-sequence-06.txt
> > Status:
> https://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/
> > Htmlized:
> http://tools.ietf.org/html/draft-ietf-json-text-sequence-06
> > Diff:
> http://www.ietf.org/rfcdiff?url2=draft-ietf-json-text-sequence-06
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: John Cowan <[email-masked]>
Date: Sat, 23 Aug 2014 14:15:47 -0400

> do this?  I can’t perceive any benefits.

It was there indeed.  IMO the RS after the text is more important than
the RS before the text: the beginning is unambiguous, but if the last
part of the sequence is "123", you don't know if that's a real 123
or a chopped 1234.  "123<RS>" is unambiguous.

So I'd reformulate the ABNF rule to say "1*RS *(JSON-text RS)".

> produces a deterministic output, then go crazy with grep and friends.

Well, that's true, but the prettyprinter in question is 'tr -d '\012',
and it will work on sequences just as well as on single texts.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
We do, doodley do, doodley do, doodley do,
What we must, muddily must, muddily must, muddily must;
Muddily do, muddily do, muddily do, muddily do,
Until we bust, bodily bust, bodily bust, bodily bust.  --Bokonon

---

From: Nico Williams <[email-masked]>
Date: Sat, 23 Aug 2014 14:07:12 -0500

> Do we need to say this? Why?

If we don't then there will be an ambiguity for any apps that could have
a notion of "empty" JSON text, though I admit that's a very strange
thing (more null than null).

Does any harm result from having that text?

> do this?  I can’t perceive any benefits.

The ABNF implicitly allows it.  No reason not to mention it.  As John
Cowan mentions, if the last text written is a number then writing an RS
after it can help detect truncated last writes.  Maybe it should be a
MUST for numbers!

> First of all, that’s %x0A not 1A.

Oops, good catch.

> expect to get a useful result.  [...]

It's a useful search space pruning technique, since grep is faster than
parsing a text and then applying JSON-aware pattern matching.

E.g.,

$ find . -name \*.json | xargs grep -l something | xargs jq '.foo.bar | select(.baz == "something")'

For some datasets that's bound to be faster than the almost equivalent:

$ find . -name \*.json | xargs jq '.foo.bar | select(.baz == "something")'

It's not reliable, of course, since every character of "something" can
be escaped (how annoying), and writing a regexp that accounts for that
might not result in a speedup.

But I would agree to removing that text because the previous paragraph
(describing other JSON sequence schemes) should suffice.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sat, 23 Aug 2014 14:20:24 -0500

> or a chopped 1234.  "123<RS>" is unambiguous.

Right, I should probably mention this.

> So I'd reformulate the ABNF rule to say "1*RS *(JSON-text RS)".

Well, if the ABNF is to describe what the parser must deal with, then
what we have is good enough.  If it must describe what the encoder must
do then "RS JSON-text [RS]" is good enough.  We could have two rules: one
for parsers and one for encoders...  The main reason the two rules would
differ is the log application with potential for truncated writes.

> and it will work on sequences just as well as on single texts.

Well, not quite.  If you're looking for strings you have to consider
that every character in a string can be escaped, so grep'ing for strings
will not be reliable...  Though one could construct a filter just for
this purpose.

$ # jgrep being like grep but with JSON escape awareness
$ find . -name \*.json | xargs jgrep -l something | xargs jq '.foo.bar | select (.baz == "something)'

Looking for numbers gets... worse.

There are other text tools that are useful.  For example: $PAGER,
$EDITOR (it might have a JSON pretty printer, but it might not handle
multi-megabyte lines well).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sat, 23 Aug 2014 18:25:00 -0500

On Sat, Aug 23, 2014 at 10:58:12AM +0200, Dominik Tomaszuk wrote:
> >[...]
> Here you can process to the first delimiter [RS], or move on and
> process to the first delimiter [US] as a group A [RS] B.

To me that's structure like this:

RS [A, B] RS C RS D

It's one-level streamed hierarchy, but it's still a hierarchy.  If
that's what you need then you should use a streaming parser.

This format is a happy middle between "streaming without a streaming
parser" and "streaming parser".  If you need to go beyond this then
IMO the trade-off is not worthwhile anymore.

> various factors, i.e. how large is logfile.)

Hmm, maybe I didn't understand the proposal.  I don't understand the A
or A and B part.

> >implement.  It'd be easier to implement an incremental, streaming JSON

Even two levels is too complex.  One level (this format) is just right.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Sat, 23 Aug 2014 21:13:57 -0400

> differ is the log application with potential for truncated writes.

My point is rather stronger:  I don't see any benefit from a leading RS,
whereas there is substantial benefit for a trailing RS.  In Tim Bray's
terms, I am proposing changing from:

for all JSON texts:
  write RS
  write text

to:

for all JSON texts:
  write text
  write RS

> will not be reliable...  

I don't understand what you are saying.  I'm saying that to make a JSON
text live on a single line, you strip all existing newline characters
and add one newline character to the end.  Period.  There are no newline
characters in JSON strings.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Eric Raymond is the Margaret Mead of the Open Source movement.
          --Bruce Perens, a long time ago

---

From: Dominik Tomaszuk <[email-masked]>
Date: Sun, 24 Aug 2014 11:19:33 +0200

> that's what you need then you should use a streaming parser.

Not exactly. When A and B are generated you can not know whether is this 
group will be large or not. In my example decide that at the end. In 
youe example I should decide at the begining. If it's large I generate US.
During processing of this I can give value specifying the limit. 
Consequently, I take A and B (if size(A and B) < limit) or only A (if 
size(A and B) > limit).

> Even two levels is too complex.  One level (this format) is just right.

Probably my proposal is more complex. But supporting of 4 separators 
from C0 can be optional, i.e.:
A RS B US C RS D
It can be processing like A RS B RS C RS D if the parser is simpler or 
as above if the parser is more advanced.

In my opinion adding of additional separators will not significantly 
affect the spec. But it gives a more accurate processing.

Dominik Tomaszuk
Research Fellow
University of Bialystok
Poland

---

From: Tim Bray <[email-masked]>
Date: Sun, 24 Aug 2014 10:16:54 -0700

For regular expressions, I still kind of like (to use normal not RFC
syntax):

(RS JSON-text)*

The leading RS might be a useful signal, and distinguishing end-of-message
is a problem solved at another level.

---

From: John Cowan <[email-masked]>
Date: Sun, 24 Aug 2014 13:23:43 -0400

> is a problem solved at another level.

It's not about end-of-message as such, it's about an unambiguous signal
that the message is complete.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Why are well-meaning Westerners so concerned that the opening of a
Colonel Sanders in Beijing means the end of Chinese culture? [...]
We have had Chinese restaurants in America for over a century,
and it hasn't made us Chinese.  On the contrary, we obliged the Chinese
to invent chop suey.            --Marshall Sahlins

---

From: Tim Bray <[email-masked]>
Date: Sun, 24 Aug 2014 10:32:59 -0700

Oh, I see... if you see "1234321<EOF>" you don’t know if it’s an error.
 OK, good argument.  So (JSON-text RS)*

---

From: Stefan Drees <[email-masked]>
Date: Sun, 24 Aug 2014 19:39:52 +0200

"I am missing a bit focus" RS 42 RS 3.1415

Should be a valid sequence of the 3 JSON texts:
["I am missing a bit focus", 42, 3.1415] here wrapped
in a JSON (heterogeneous) array.

I would expect the producer might be as simple as in python eg.
'\x1E'.join(json_texts)

The consumer of such a snippet should be configured to either trust
any info part as complete or to well, do what? Force the producer to add 
an RS always? ... In my daily diff business on source code, missing 
trailing newlines are not rare and I doubt we will enforce this here for 
good.

The start of message might always be eagerly prefixed with an RS if
the consumer absolutely needs it. In python you often use StringIO to 
fake a prefix dummy section so you can consume and produce ini files 
with a global "non-section".

I ssupsect we - whilst drifting away from the well-known newline as 
separater of log entries to the RS regime - lose a bit focus about what 
we expect to be placed where by whom and what for.

{"top-postingly-yours": "Stefan"}

Am 24.08.14 19:16, schrieb Tim Bray:
> The leading RS might be a useful signal, and distinguishing
> end-of-message is a problem solved at another level.
>
>
> On Sat, Aug 23, 2014 at 6:13 PM, John Cowan <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Nico Williams scripsit:
>
>     <mailto:[email-masked]>

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 25 Aug 2014 09:55:37 -0400

Hi Tim,

--On August 24, 2014 at 10:32:59 AM -0700 Tim Bray <[email-masked]> 
wrote:

> Oh, I see... if you see "1234321<EOF>" you don’t know if it’s an
> error.  OK, good argument.  So (JSON-text RS)*
>

Isn't one the goals here to allow any number of processes to potentially 
log to the same file without having to check that the previous entry ended 
with RS (the assumption being that a single file.write() is atomic)? If so, 
then always pre-pending the entries with RS guarantees that even partially 
written entries are properly delineated.

-- 
Cyrus Daboo

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 25 Aug 2014 15:52:40 +0000

>terms, I am proposing changing from:

The other use case to keep in mind is concatenating multiple 
possibly-truncated files together.  With the leading RS, you're always 
sure that the first (presumably well-formed) JSON-text in the second file 
can be read.

RS (JSON-text RS)*

Seems to be a good compromise, except for Cyrus's use case of multiple 
writers.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Mon, 25 Aug 2014 12:03:03 -0400

> sure that the first (presumably well-formed) JSON-text in the second file 

Excellent point, and a good grammar rule.  I tried to design a rule that
would allow zero-or-more RSes at the beginning and end with one-or-more
in between and still allow an empty sequence, but found it too difficult.
As a compromise with the original design, I suggest:

    1*RS *(JSON-text 1*RS)

That guarantees delimiters at both ends, makes the minimal empty sequence
simply RS RS, and allows Bray pseudo-code like this:

For each JSON text, atomically:
  write RS
  write the JSON-text
  write RS

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
I Hope, Sir, that we are not mutually Un-friended by this Difference
which hath happened betwixt us.
     --Thomas Fuller, Appeal of Injured Innocence (1659)

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 25 Aug 2014 16:15:33 +0000

>For each JSON text, atomically:

Wouldn't that be:

*(1*RS JSON-text 1*RS)

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Mon, 25 Aug 2014 12:22:09 -0400

> *(1*RS JSON-text 1*RS)

That rule requires, rather than simply permitting, the pattern
RS-text-RS-RS-text-RS.  My version also permits that, but further permits
RS-text-RS-text-RS.  It also guarantees that all sequences, even empty
ones, begin and end with RS (whereas in your version the empty sequence
is the empty body).

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
I don't know half of you half as well as I should like, and I like less
than half of you half as well as you deserve.  --Bilbo

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 25 Aug 2014 16:35:43 +0000

>ones, begin and end with RS (whereas in your version the empty sequence

+1.  Reminder, then, John's version was:

1*RS *(JSON-text 1*RS)

And the generator rule was:

I'd suggest a slight modification in that if you're not writing bare 
numbers as your JSON-texts, the second RS isn't useful and could be 
skipped.  If most folks are logging objects, you get to save LOTS OF BYTES 
(not).  However, it's one less thing to get into the atomic write, which 
might be interesting for someone.

Do you think something like Tim's generator rule would be useful to go 
into the document as implementation guidance?

-- 
Joe Hildebrand

---

From: "Manger, James" <[email-masked]>
Date: Tue, 26 Aug 2014 10:27:34 +1000

> 1*RS *(JSON-text 1*RS)

This looks awkward. It doesn't allow a newline at the end of the file. It is hard to avoid a trailing newline if you edit a file with many text editors. We will be searching stackoverflow for how to remove trailing newlines when our JSON-text-sequence tools fail.
This is also likely to create very long lines of text (eg no newlines in a whole log), which can be annoying.
It doesn't allow leading whitespace or whitespace between entries, which you often get inadvertently from web templating schemes (eg .jsp files).

--
James Manger

---

From: John Cowan <[email-masked]>
Date: Tue, 26 Aug 2014 00:11:32 -0400

Manger, James scripsit:
> > 1*RS *(JSON-text 1*RS)
> 
> This looks awkward. It doesn't allow a newline at the end of the
> file. It is hard to avoid a trailing newline if you edit a file with
> many text editors. We will be searching stackoverflow for how to remove
> trailing newlines when our JSON-text-sequence tools fail.

Eh, it's unquoted whitespace.  No reason to choke on it.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
                I am a member of a civilization. --David Brin

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Aug 2014 00:09:05 -0500

I don't think we need to cover the incomplete JSON text case with
ABNF.  Obviously that would be a lot of work; we'd most likely
handwave around it anyways.

My proposal is that we RECOMMEND or REQUIRE (in prose) that the
encoder follow any top-level numbers with either a ws or an RS -- they
MAY also do it for all texts if that's easier.  Then we should leave
the ABNF as-is in -06.  Extra RSes are obviously fine.

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Tue, 26 Aug 2014 15:11:52 +1000

>> This looks awkward. It doesn't allow a newline at the end of the file. 
>> It is hard to avoid a trailing newline if you edit a file with many 
>> text editors. We will be searching stackoverflow for how to remove 

> Eh, it's unquoted whitespace.  No reason to choke on it.

Whitespace is allowed as part of JSON-text, but this ABNF doesn't allow it beyond the RS boundaries.

The following 16-byte file would fail:

<RS>123<RS>\n
<RS>{"a":2}<RS>\n

--
James Manger

---

From: Stefan Drees <[email-masked]>
Date: Tue, 26 Aug 2014 07:30:32 +0200

> The following 16-byte file would fail:

nice example IMO that could easily come out of a robust multi-producer 
setting.

Stefan\n

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Aug 2014 00:48:58 -0500

> The following 16-byte file would fail:

Why would it?  The I-D says that incomplete/invalid texts SHOULD NOT
result in failure.  We don't have write an ABNF rule for
invalid-JSON-text since it's obviously anything (other than RS!) that
doesn't match JSON-text.

(We could write a rule for invalid-JSON-text: it'd be a sequence 1* of
any bytes other than RS!  But this seems tedious because no one is going
to write a parser (or encoder) only from the ABNF.)

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Tue, 26 Aug 2014 16:28:34 +1000

> Why would it?  The I-D says that incomplete/invalid texts SHOULD NOT result in failure.  We don't have write an ABNF rule for invalid-JSON-text since it's obviously anything (other than RS!) that doesn't match JSON-text.

But I don't want a trailing newline treated as an incomplete JSON text that indicates a write was truncated. I don't want this example reported to the app as: 4 entries; [number, invalid, object, invalid].

--
James Manger

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 28 Aug 2014 19:05:37 +0900

> As much as it is fun to keep debating old topics, it would help the WG if folks could focus on the current work. We would like to finish up, and the following WG LC is a way for us to determine how…

Given the position I have expressed for the previous last call (and/or 
earlier), I haven't reread the latest draft because I have absolutely no 
indication that my position would change.

Also, there is still a lack of confirming comments (something along the 
lines of 'I read it - please ship it'). Even the above repeated begging 
from Paul to help the chairs determine consensus has only let to more 
bikeshedding.

Given this, I see ample (not explicit but implicit) support for my 
position that trying to spec "JSON text sequences" has failed. The 
format simply doesn't provide enough differentiation from a JSON array 
(*), and too much possibility for useless variation.

Regards,   Martin.

(*) Requiring RS at the start and the end of the whole thing comes 
terribly close to '[' and ']' at the beginning and the end of a JSON array.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 28 Aug 2014 17:13:06 +0000

>from Paul to help the chairs determine consensus has only let to more 

In this case, one person's bikeshed is another person's technical concern. 
Once we get the ABNF straightened out, I'm a strong +1 on publishing this 
document.

>terribly close to '[' and ']' at the beginning and the end of a JSON 

Not that close, because you can have lots of extra RS's in a document, and 
it will still parse.  Any stray ']' will break an array.  I've written the 
code for both JSON arrays and XML files trying to keep them well-formed 
will appending.  The code is error-prone and will never work with multiple 
writers.

-- 
Joe Hildebrand