Subject: [Json] discoverability of separators in draft-ietf-json-text-sequence

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 4 Aug 2014 19:23:40 +0000

What if you could use basically any separator you wanted in a text
sequence, every sequence MUST start with that separator, and we gave
suggestions for ones we think would work with some text about the
trade-offs?

Processors would read up to the first [, {, ", n, t, f, ., -, or number,
and treat that as the separator (potentially ignoring an irrational BOM if
they like).  The separator MUST be escaped in any of the JSON texts in the
sequence.  Whenever the separator is read, a new JSON text starts
immediately after it.  Any pending text that was not (yet) a valid JSON
text that was waiting to be processed when the separator is encountered
signifies a write error, and MAY be ignored.  Whitespace could trail a
JSON text before the next separator, which could be ignored according to
the previous rule.

ABNF such as:

JSON-sequence = *(sep value ignorable-ws)
sep = 1*(not [{"ntf.-0123456789)
ignorable-ws = ws
ws = <given by RFC7159>
     value = <given by RFC7159>

The other syntactically valid characters in a JSON text should probably be
forbidden in a separator as well.

Alternately, we could decide that the separator was always exactly one
byte, but keep everything else the same as above, simplifying the
processors a good amount.

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Mon, 4 Aug 2014 15:27:32 -0400

> forbidden in a separator as well.

Essentially every character is syntactically valid in a JSON text, excepting
C0 controls, which is why we have been talking about newline and RS.

> processors a good amount.

That limits it to ASCII, which I suppose is tolerable.
But at that rate you might as well just pick one, like "/".

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Would your name perchance be surname Puppet, given name Sock?
                --Rick Moen

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 4 Aug 2014 19:34:32 +0000

>C0 controls, which is why we have been talking about newline and RS.

I'm only interested in the ones that can't be escaped, such as ",".  If
you picked "g" as your separator, you could get by, as long as all of your
strings used "\u0067".

>But at that rate you might as well just pick one, like "/".

Good point.  It would also be a lot easier to write the ABNF as just any
C0 control.

It's also likely worth mentioning as implementation advice that you might
get a stream of octets that are invalid UTF-8 (such as sep 22 F0 9F 92 sep
good-value), and a processor should still be able to recover when it sees
the separator.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 4 Aug 2014 12:38:14 -0700

<no hat>

> suggestions for ones we think would work with some text about the

That would preclude starting the value with ignorable-ws, while allowing the value to have ignorable-ws after. Allowing ignorable-ws in one of the two positions seems likely to cause errors.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 4 Aug 2014 22:07:38 +0200

On 04 Aug 2014, at 21:23, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> What if you could use basically any X you wanted in a Y,

That is a good approach if there is an installed base of Y with many different existing choices for X and there is a benefit to be compatible with all/many of them.

For something designed from scratch, I would always try to make a decision.
(At least when it comes to bikeshed-level decisions.)

(Sometimes the decision turns out to be the subtly wrong one after a couple decades, as with assigning the negative sign to the flow of electrons or with MSB-first network byte order.  Letting textbooks choose whether they want electrons to be negative in the preface sounds about as bad as having a protocol choose byte order in the preamble.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 4 Aug 2014 15:18:42 -0500

> suggestions for ones we think would work with some text about the

I really like a variant of this.  See below.

> Processors would read up to the first [, {, ", n, t, f, ., -, or number,

It's simpler than that!

The sequence parser state machine would work as follows:

0) the sequence parser has two state variables: one referencing a text
   that has been parsed but not yet output, and one referencing the text
   currently being parsed.

1) as soon as a text parse begins (with _any_ character), the previous
   parsed text (if any) is output and the parsed-but-not-yet-output
   state variable is reset.

   conversely, when a text parse completes the text is placed in the
   parsed-but-not-yet-output state variable and the current text
   variable is reset.

   however, a text consisting of a top-level number, boolean, or null,
   does not complete parsing until a suitable explicit separator is
   received, or a ", [, or { is received (no ambiguity == separator not
   needed).

2) if any character that is suitable for use as a text separator is
   received then:

    - if it is a JSON whitespace character then it is part of the
      current/next text to be parsed (triggering (1) if it is the first
      character after the last text)

    - else the current text is discarded as incomplete and (1) is
      triggered.

This makes Riak's use of JSON text sequences (sequences of objects with
no separators) valid sequences as well (which, incidentally, jq can
handle today).  It also very closely matches what jq's parser does.

Encoders must pick a separator suitable to their needs.  Loggers can
either RS (or BEL, or ...) or they can s/LF/SP/ then use LF.

I like this a lot because it's easy to implement and allows the use of
any ASCII control character, including whitespace, as a separator.  It
also allows the use of no separator when one is not needed.

I'd be very happy with this!  It's the jq sequence encoder where I
really don't want to use RS (or anything other than LF).  The parser can
easily deal with a variety of separators.

(jq's parser can handle missing separators between arrays, strings, and
objects, but not between booleans, nulls, or numbers.  The encoder
always emits a newline (LF) separator.  Making the parser handle more
separators would be trivial.)

For example:

$ # Usage is: jq <filter>
$ #
$ # The '.' filter means: copy input stream to output.
$
$ # Example 1: with SP as separator only where none is needed to
$ # disambiguate.
$ #
$ echo '1"foo"[]{}true null false'|jq .
1
"foo"
[]
{}
true
null
false
$
$ # Example 2: with no separator where one IS needed to disambiguate.
$ #
$ for i in '1true' 'true2' 'truefalse' 'nullfalse' 'null3' '4null'; do
echo "$i" | jq .; done
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
$ 

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Mon, 4 Aug 2014 13:21:56 -0700

I fail to perceive any benefit in allowing the use of more than one unique
ASCII control character as a separator.

On Mon, Aug 4, 2014 at 1:18 PM, Nico Williams <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Nico Williams <[email-masked]>
Date: Mon, 4 Aug 2014 15:26:57 -0500

On Mon, Aug 04, 2014 at 10:07:38PM +0200, Carsten Bormann wrote:
> That is a good approach if there is an installed base of Y with many
> different existing choices for X and there is a benefit to be
> compatible with all/many of them.

And... there is!

$ # Usage is: jq <filter>
$ #
$ # The '.' filter means: copy input stream to output.
$
$ # Example 1: with SP as separator only where none is needed to
$ # disambiguate.
$ #
$ echo '1"foo"[]{}true null false' | jq .
1
"foo"
[]
{}
true
null
false
$
$ # Example 2: with no separator where one IS needed to disambiguate.
$ #
$ for i in '1true' 'true2' 'truefalse' 'nullfalse' 'null3' '4null'; do echo "$i" | jq .; done
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
$ 
$ # Example 3: use LF as separator
$ #
$ printf '1\n"foo"\n[]\n{}\ntrue\nnull\nfalse' | jq .
1
"foo"
[]
{}
true
null
false
$ 

https://stedolan.github.io/jq
https://github.com/stedolan/jq

Riak uses JSON text sequences without any separator (the texts it
outputs are objects at the top-level, so there's no need to
disambiguate).

> For something designed from scratch, I would always try to make a
> decision.  (At least when it comes to bikeshed-level decisions.)

I agree.

> (Sometimes the decision turns out to be the subtly wrong one after a
> couple decades, as with assigning the negative sign to the flow of
> electrons or with MSB-first network byte order.  Letting textbooks
> choose whether they want electrons to be negative in the preface
> sounds about as bad as having a protocol choose byte order in the
> preamble.)

I agree, though those are examples where either option was just as good
as the other and the decision to be made perfectly arbitrary :)

Nico
--

---

From: Stephen Dolan <[email-masked]>
Date: Tue, 5 Aug 2014 15:17:11 +0100

> suggestions for ones we think would work with some text about the

Concatenating two well-formed logfiles gives a well-formed logfile.
This property is true of almost every logfile format, widely relied
upon by logfile processing tools, and broken by the proposed format.

Stephen

---

From: Matthew Morley <[email-masked]>
Date: Tue, 5 Aug 2014 11:31:13 -0400

> upon by logfile processing tools, and broken by the proposed format.

+1

-- 
Matthew P. C. Morley

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Aug 2014 10:38:35 -0500

+1

Thanks for capturing that point.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 5 Aug 2014 16:27:46 +0000

>upon by logfile processing tools, and broken by the proposed format.

Understood.  For the people that didn't follow the subtlety, each file
might use a different separator.

We need to pick just one if this use case is important.

-- 
Joe Hildebrand