Subject: [Json] json-text-sequence-05

From: Nico Williams <[email-masked]>
Date: Wed, 20 Aug 2014 01:32:30 -0500

----- Forwarded message from [email-masked] -----

Date: Tue, 19 Aug 2014 23:30:51 -0700
From: [email-masked]
To: Nico Williams <[email-masked]>, Nicolas Williams <[email-masked]>
Subject: New Version Notification for draft-ietf-json-text-sequence-05.txt

A new version of I-D, draft-ietf-json-text-sequence-05.txt
has been successfully submitted by Nicolas Williams and posted to the
IETF repository.

Name:		draft-ietf-json-text-sequence
Revision:	05
Title:		JavaScript Object Notation (JSON) Text Sequences
Document date:	2014-08-20
Group:		json
Pages:		9
URL:            http://www.ietf.org/internet-drafts/draft-ietf-json-text-sequence-05.txt
Status:         https://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/
Htmlized:       http://tools.ietf.org/html/draft-ietf-json-text-sequence-05
Diff:           http://www.ietf.org/rfcdiff?url2=draft-ietf-json-text-sequence-05

Abstract:
   This document describes the JSON text sequence format and associated
   media type.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

----- End forwarded message -----

---

From: Nico Williams <[email-masked]>
Date: Wed, 20 Aug 2014 01:38:52 -0500

The RS version is remarkably simpler.  I still prefer the LF version
for the reasons I've given before, but I can't deny that the RS
version is appealing.

Nico
--

---

From: Dominik Tomaszuk <[email-masked]>
Date: Wed, 20 Aug 2014 13:43:51 +0200

RS is not defined in RFC5234.

Best,
Dominik

---

From: Tim Bray <[email-masked]>
Date: Wed, 20 Aug 2014 07:48:03 -0700

Comments on this draft:

Section 1.1. Don’t think this section is necessary any more
Section 2. Remove “, which may be delimited by an external end-of-stream
marker)” which adds no value. I have no idea what an “end-of-stream marker”
means.
Section 2.2 Not sure about this section; this document exists to describe
the RS-delimited version of multiple-JSON-texts; the fact that there are a
number of ways to do this is interesting, but that’s not what this exists
to describe.  I get it, Postel’s law and all that, but it feels like a
distraction for the implementor.

Wow, is this ever simple.  I like it and I think most implementors would
too.

On Wed, Aug 20, 2014 at 4:43 AM, Dominik Tomaszuk <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 20 Aug 2014 08:15:12 -0700

> RS is not defined in RFC5234.

Correct.

Nico: for the -06, please add the following to the ABNF:

  RS = %x1E

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 20 Aug 2014 08:30:06 -0700

<No hat>

> Section 1.1. Don’t think this section is necessary any more

Agree.

> Section 2. Remove “, which may be delimited by an external end-of-stream marker)” which adds no value. I have no idea what an “end-of-stream marker” means.

Agree.

> Section 2.2 Not sure about this section; this document exists to describe the RS-delimited version of multiple-JSON-texts; the fact that there are a number of ways to do this is interesting, but that’s not what this exists to describe.  I get it, Postel’s law and all that, but it feels like a distraction for the implementor.

Disagree; I think it is reasonable to leave it in.

> Wow, is this ever simple.  I like it and I think most implementors would too.

+1.

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Wed, 20 Aug 2014 12:05:19 -0400

> distraction for the implementor.

-1.  Whitespace-separated sequences should be understood for backward
compatibility.

The sentence "JSON text sequence encoders MAY emit an RS after emitting
a JSON text" should instead be encoded in the BNF by adding "*RS" to
the end of the JSON-sequence rule.

> Wow, is this ever simple.  I like it and I think most implementors would

+1

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Nobody expects the RESTifarian Inquisition!  Our chief weapon is
surprise ... surprise and tedium  ... tedium and surprise ....
Our two weapons are tedium and surprise ... and ruthless disregard
for unpleasant facts....  Our three weapons are tedium, surprise, and
ruthless disregard ... and an almost fanatical devotion to Roy Fielding....

---

From: "Manger, James" <[email-masked]>
Date: Thu, 21 Aug 2014 14:26:48 +1000

http://tools.ietf.org/html/draft-ietf-json-text-sequence-05

>> Wow, is this ever simple.  I like it and I think most implementors would too.

Simplicity is nice, but in this case it doesn’t make it usable.

For instance, a major failing of this RS-based version is that a single JSON value is no longer a valid JSON Sequence of 1 item (since it doesn’t have a leading RS).

Being simple to implement a JSON Sequence parser is nice, but that consideration should be a distant second to being simple to create JSON Sequences from any of our text-based tools. RS really hurts the latter. I don’t know of a single format that uses RS as a separator; compared to a squillion that use newline.

--
James Manger

---

From: Stefan Drees <[email-masked]>
Date: Thu, 21 Aug 2014 07:08:28 +0200

>>>Wow, is this ever simple.  I like it and I think most implementors would
+0 (being relevant once in many use cases, 'til the nut is cracked)

> For instance, a major failing of this RS-based version is that a single
> JSON value is no longer a valid JSON Sequence of 1 item (since it
> doesn’t have a leading RS).
>
> Being simple to implement a JSON Sequence parser is nice, but that
> consideration should be a distant second to being simple to create JSON
> Sequences from any of our text-based tools. RS really hurts the latter.
> I don’t know of a single format that uses RS as a separator; compared to
> a squillion that use newline.

+1
"stefan"

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Aug 2014 00:28:06 -0500

> Simplicity is nice, but in this case it doesn’t make it usable.

That's my take.  I don't see how I can use JSON text sequences with RS
-- if that were the consensus we might yet publish, but I would
probably not implement an encoder for it.

Among the many problems with using RS is the same problem some of us
had with the I-JSON requirement that the top-level be an object: it
doesn't compose.

Consider jq for instance.  It uses LF, and will continue to regardless
of what we choose here, though I would add an option for emitting RS
instead.  The problem is that the point in a "pipeline" at which the
network intrudes and demands RS is not necessarily clear, but at all
other points existing tooling won't handle RS.  Net effect: jq will be
harder to use.

> For instance, a major failing of this RS-based version is that a single JSON
> value is no longer a valid JSON Sequence of 1 item (since it doesn’t have a
> leading RS).

Indeed!

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 07:15:52 -0700

> For instance, a major failing of this RS-based version is that a single JSON value is no longer a valid JSON Sequence of 1 item (since it doesn’t have a leading RS).

What scenarios do you envision where there would be a JSON sequence with only a single entry?

> Being simple to implement a JSON Sequence parser is nice, but that consideration should be a distant second to being simple to create JSON Sequences from any of our text-based tools. RS really hurt…

If you disagree with the text at the beginning of Section 1. However, that set of scenarios is the one we are evaluating the document against.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 21 Aug 2014 07:56:13 -0700

On Wed, Aug 20, 2014 at 9:26 PM, Manger, James <
[email-masked]> wrote:

> JSON value is no longer a valid JSON Sequence of 1 item (since it doesn’t
> have a leading RS).
>

​Why is this a failing:  Code to create a seq:

foreach record
  generate JSON
  emit 0x1E
  emit JSON

Code to read a seq:

while !eof
  scan for 0x1E
  parse JSON​
​​
​I can’t imagine how implementers will get this wrong
​

> Sequences from any of our text-based tools. RS really hurts the latter. I
> don’t know of a single format that uses RS as a separator; compared to a
> squillion that use newline.
>

​I can’t imagine wanting to generate a seq from anything other than the
logging module of a computer program.  See above.​

---

From: Tim Bray <[email-masked]>
Date: Thu, 21 Aug 2014 08:00:24 -0700

> had with the I-JSON requirement that the top-level be an object: it

​Excuse me… (0x1e json-text​)* composes beautifully.
​​

> of what we choose here,

If a little-used piece of software announces in advance it’s going to
ignore the output of a WG, that’s worth consideration, but in this case it
doesn’t seem like a very decisive argument.

> though I would add an option for emitting RS
> network intrudes and demands RS is not necessarily clear,

​Yes it is. When you want to emit a seq, you prefix each text with 0x1E.
 When you want to read it, you scan for and consume the 0x1E’s, and pass
the texts in between to the application software.

It seems that this is a reliable, simple, and no-brainer for developers to
implement.

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Aug 2014 11:42:12 -0500

> [...]

It's the inspiration for this I-D.  That's not nothing.  It's running
code, for one.

> doesn’t seem like a very decisive argument.

Tim, jq deals with text, because JSON is text.  Except at the network
boundary it would have to continue to use text for backwards
compatibility reasons.

The question is: how to make the network boundary (where we'd have to
use RS) clear easily.

I can make a good argument for why this isn't a problem: suppose we were
to add CBOR (or some other binary JSON encoding) support...  And/or a
variety of other JSON-model-compatible encodings.  Then clearly the user
would have to demarcate the network boundary.  Therefore using RS at the
network boundary is no more burdensome than supporting any other
encoding.

(I thought of that last night, and it's almost enough to convince me.
I'll sleep on it some more.  But again, there's no chance that I'd use
RS except at the network boundary, because LF is just too darned
convenient/useful/useable.)

Also, I think I'd recommend that texts be followed by LF in addition to
being prefixed by RS: that way text-oriented tools that ignore RS will
continue to work.  Maybe _that_ will be good enough, if enough tools
ignore RS!  That would bring me over.

> Yes it is. When you want to emit a seq, you prefix each text with 0x1E.

jq has always dealt with sequences of JSON texts.

> the texts in between to the application software.

Parsing it is no big deal.  It's knowing when to use LF (because
backwards compat) and when to use RS (because interop).

> It seems that this is a reliable, simple, and no-brainer for developers to

The LF thing is also simple to implement.  And if loggers remove
internal LFs then it's also reliable.  It's as simple and reliable as
RS.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Thu, 21 Aug 2014 09:57:33 -0700

> internal LFs then it's also reliable.  It's as simple and reliable as

​If they have to remove internal LFs to make it reliable, then by
definition it’s not as simple as RS, which works with any existing JSON
text, and with any existing JSON generation software, without modification.​

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Aug 2014 12:00:08 -0500

I'll submit a -06 tonight.

I think I'm won over to the RS side, but with a caveat: I'd like to
recommend that JSON texts in the sequence also be followed by LF, since
that seems to make it possible to use JSON text sequences with
line-oriented tools that ignore RS.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 10:08:32 -0700

> I'll submit a -06 tonight.

Thank you.

> line-oriented tools that ignore RS.

No: that is an inappropriate change at this time. The document states that it is for log-style sequences, not line-oriented tools that are not parsing JSON texts correctly.

Please make the few changes we asked for so we can start another WG Last Call.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 21 Aug 2014 10:22:53 -0700

On Thu, Aug 21, 2014 at 10:00 AM, Nico Williams <[email-masked]>
wrote:

>  I'd like to

​What’s an example of such a tool?  The JSON parsers I know would report a
syntax error upon encountering a floating 0x1E (one of the reasons I like
it).​

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Aug 2014 13:04:29 -0500

On Thu, Aug 21, 2014 at 10:22:53AM -0700, Tim Bray wrote:
> What’s an example of such a tool?  The JSON parsers I know would report a
> it).

I'm referring to things like grep, wc, ...

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 21 Aug 2014 20:00:03 +0000

>I'm referring to things like grep, wc, ...

Clarifying for Nico here.  You'd like to be able to grep for a value in a 
stream, and have just the matching line returned rather than the whole 
file (better yet, you might want the entire matching record...).  With wc, 
you might like the number of lines to mean something with respect to the 
number of records.  Without a newline, both of those use cases fail out of 
the box.

Note, you can use awk to fix a bunch of these use cases:

awk -v RS=$'\x1e' '/pattern/' file

would return all of the records matching pattern in file, and:

awk -v RS=$'\x1e' 'END{print NR-1}' file

would print the number of records.  Note: awk uses whatever string or 
regex is in RS as the record *separator*, not as a record start or end, 
which is why you have to subtract one.  Further note: it's amusing that 
setting RS=(ASCII RS) is so syntactically difficult.

Since JSON-text includes ws in 7159, you MAY already include a linefeed 
after each record with the -05 draft.  We could potentially choose to give 
implementation advice without changing any of the normative text.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Aug 2014 15:46:18 -0500

> Clarifying for Nico here.  You'd like to be able to grep for a value in a 

Thanks :)  Yes, that's it.

> awk -v RS=$'\x1e' '/pattern/' file

Yes, this is true.

>                                 [...].  Further note: it's amusing that 

It is.  It'd be nice to have a backslash escape for it.

> implementation advice without changing any of the normative text.

Indeed, I can.  It costs nothing to emit an extra LF, but it does help.
I'd not make a requirement, of course, but I'd recommend it.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 21 Aug 2014 15:51:05 -0700

> I'd not make a requirement, of course, but I'd recommend it.

Please be careful with your wording. This is not an "extra" LF if there is no LF there already. I believe you are talking about recommending (not requiring) that every JSON-text have a LF character as the last character in the second ws of "ws value ws", yes? However, if the first ws has a LF, or if the second ws has a LF that is not the last one, for any of the JSON-texts in the sequence, it kind of scorches any advantage of your suggestion. So, if you really mean "all initial ws's should have no LF, and all terminal ws's should have exactly one LF, and it should be at the end", you need to say that instead.

As you tell, I am skeptical that this recommendation is going to help your use case unless the recommendation is complicated.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 21 Aug 2014 17:12:58 -0700

You know, much and all as I am an old command-line hound and use them all
the time for Apache logs, the notion of running that kind of stuff over
JSON makes me nervous, and the supposition that sticking a \n after the end
of a JSON-text is going to prevent all the things that could go wrong...
no.  I’m just not sure you can have your JSON cake and eat your
command-line goodies too.  But if people are convinced that that \n will
actually increase the usefulness, I guess it’s not actively harmful.

​​

---

From: "Manger, James" <[email-masked]>
Date: Fri, 22 Aug 2014 12:43:35 +1000

The RS-version of JSON Text Sequences may as well remove all mention of JSON.
The format is simply:
  * ( RS blob )
where blob is anything that doesn’t contain an RS byte/character.

It is nice and simple.
It is not "text" (because it isn't easy to work with in all our text tools).
You will always need a new RS-aware explicit tool/step to convert to and from the RS-prefixed format.

It has no connection to a JSON-based ecosystem.
It is not a "cow path to pave"; no one is using RS like this.

On the other hand, logging JSON objects one per line is an obvious approach. A bunch of system have done that. Such a syntax is still unambiguous even if the JSON objects are pretty-printed (have newlines inside the objects). It is still unambiguous for any JSON values (not just objects) if you add an easy constraint such as having a trailing newline. If tools take the extra effort to support this (instead of assuming one-object-per-line) because it is a spec then we can have our JSON cake and eat our command-line goodies as well. 

P.S. The RS-version should be *(RS JSON-text), not *(1*RS JSON-text). 

--
James Manger

From: json [mailto:[email-masked]] On Behalf Of Tim Bray
Sent: Friday, 22 August 2014 10:13 AM

You know, much and all as I am an old command-line hound and use them all the time for Apache logs, the notion of running that kind of stuff over JSON makes me nervous, and the supposition that sticking a \n after the end of a JSON-text is going to prevent all the things that could go wrong... no.  I’m just not sure you can have your JSON cake and eat your command-line goodies too.  But if people are convinced that that \n will actually increase the usefulness, I guess it’s not actively harmful.

---

From: Nico Williams <[email-masked]>
Date: Fri, 22 Aug 2014 01:10:26 -0500

> where blob is anything that doesn’t contain an RS byte/character.

Yes, that's true.  But this is a data format where we're defining the
blob as a JSON text :)

> It is nice and simple.

Yes, it is.  I forget who first suggested RS (searching the archives).

> It is not "text" (because it isn't easy to work with in all our text tools).

Some text tools seem to do "the right thing" as to the RS.

> You will always need a new RS-aware explicit tool/step to convert to and from the RS-prefixed format.

Generally, yes.

> It has no connection to a JSON-based ecosystem.

I disagree.

> It is not a "cow path to pave"; no one is using RS like this.

No one is using RS, most likely.

> On the other hand, logging JSON objects one per line is an obvious approach. A bunch of system have done that. Such a syntax is still unambiguous even if the JSON objects are pretty-printed (have n…

Yes.

> P.S. The RS-version should be *(RS JSON-text), not *(1*RS JSON-text).

If the application is a log application then it has to be something
like *(RS (JSON-text / invalid-JSON-text)).  I'm not up for writing a
syntax for invalid JSON texts though!  :)

This is why I settled for *(1*RS JSON-text), but yes, you're right:
encoders MUST emit a single RS per-text, and parsers SHOULD discard
empty/invalid texts.

How would one express that in ABNF?

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Fri, 22 Aug 2014 13:31:39 -0700

> where blob is anything that doesn’t contain an RS byte/character.

The other example I can think of such a format is: XML.  Hmmmmmmmm

> On the other hand, logging JSON objects one per line is an obvious
> approach. A bunch of system have done that. Such a syntax is still
> unambiguous even if the JSON objects are pretty-printed (have newlines
> inside the objects). It is still unambiguous for any JSON values (not just
> objects) if you add an easy constraint such as having a trailing newline.
> If tools take the extra effort to support this (instead of assuming
> one-object-per-line) because it is a spec then we can have our JSON cake
> and eat our command-line goodies as well.
>

---

From: Nico Williams <[email-masked]>
Date: Fri, 22 Aug 2014 15:43:49 -0500

> The other example I can think of such a format is: XML.  Hmmmmmmmm

We could produce another MIME type for that :)

We could do it now (since it should be trivial, right?  (I assume RS
can't appear in a proper XML doc, even as CDATA??)  Or we could leave it
for the first person to need it.

Comments?

Nico
--

---

From: John Cowan <[email-masked]>
Date: Fri, 22 Aug 2014 17:00:05 -0400

> for the first person to need it.

Right, neither in XML 1.0 nor XML 1.1.  But this isn't the right place.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
All Gaul is divided into three parts: the part that cooks with lard and goose
fat, the part that cooks with olive oil, and the part that cooks with butter.
  --David Chessler

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 22 Aug 2014 21:08:00 +0000

On 8/21/14, 10:51 PM, "Paul Hoffman" <[email-masked]> wrote:

>Please be careful with your wording. This is not an "extra" LF if there 
>is no LF there already. I believe you are talking about recommending (not 
>requiring) that every JSON-text have a LF character as the last character 
>in the second ws of "ws value ws", yes? However, if the first ws has a 
>LF, or if the second ws has a LF that is not the last one, for any of the 
>JSON-texts in the sequence, it kind of scorches any advantage of your 
>suggestion. So, if you really mean "all initial ws's should have no LF, 
>and all terminal ws's should have exactly one LF, and it should be at the 
>end", you need to say that instead.

I don't think so.  If the use case is "I want to use grep, and not have 
the whole file show up when the word I'm searching for matches", then any 
number of LF's before or after the JSON-text fixes the problem.

>As you tell, I am skeptical that this recommendation is going to help 
>your use case unless the recommendation is complicated.

The recommendation could be "throw a LF in every once in a while if you 
want grep to be more easily useful, but note that they are completely 
semantically uninteresting", which doesn't seem very complicated.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Fri, 22 Aug 2014 16:14:24 -0500

On Fri, Aug 22, 2014 at 09:08:00PM +0000, Joe Hildebrand (jhildebr) wrote:
> >is no LF there already. [...]
> 

In the end I only added informative text about this.  It should be good
enough and unobjectionable (right?).

Nico
--