Subject: [Json] draft-ietf-json-text-sequence-01 comments

From: "Manger, James" <[email-masked]>
Date: Fri, 9 May 2014 18:03:28 +1000

Comments on draft-ietf-json-text-sequence-01

Section 2.

The reason to NOT REQUIRE a newline: even if the spec says newlines are required some implementations may accept other sequences when they are unambiguous (such as {"a":1}{"b":2}); some people will rely on that leniency and omit newlines; leading to interop problems with implementations that require newlines as per the spec.

A reason to REQUIRE a newline: it makes it feasible to find the start of the next JSON value from anywhere within a sequence (ie resync).

The draft 01 rule of requiring a whitespace, but not necessarily a newline, is the worst solution. It still allows interop problems if people rely on lenient implementations. It doesn’t guarantee that resync is possible.

I don’t think anyone wants to be fully lenient by accepting unambiguous sequences such as 1true4false. So I suggest we go with option B: require a newline after every JSON value.

Suggested text:

  JSON-sequence = *whitespace *(JSON-text *whitespace %x0A *whitespace)
  whitespace = %x20 / %x09 / %x0A / %x0D
  JSON-text = <given by RFC7159>

  A JSON sequence is the concatenation of zero or more JSON texts,
  each followed by a newline character.

I would drop the rest of the section (Requirements; truefalse paragraph).

Section 3. Use for logfiles

This section would be better if it focussed on being able to detect the start of the next JSON value from anywhere in a JSON sequence, ie resynchronise.

This can be useful for random access to a log file, or for recovering after an entry is corrupted (eg truncated due to a power failure part way through a write).

I would drop the details of trying to recover the 1st entry immediately after a truncated entry. I don't think it is feasible in general (eg when values are numbers); you can't even detect some truncations (eg truncating a whole entry); and the "scan backwards .. looking for a newline followed by a valid JSON text" sounds wrong as the previous newline has been truncated. Scanning back (from a boundary) a char at a time until the whole lot parses as valid JSON is probably the best you can do.

Section 4. Security considerations

The 3rd paragraph no longer applies as whitespace (or hopefully newline) is always required.

Section 5. IANA considerations

As we are specifying a media type, why not define a fragment id. For instance,
a fragment id that is a decimal integer n (matching 1*DIGIT) selects the nth entry in the sequence.

Typo: change NL to %x0A in the ABNF.
Typo: "form a logical end" to "from a logical end"

--
James Manger

---

From: Nico Williams <[email-masked]>
Date: Fri, 9 May 2014 12:21:57 -0500

> A reason to REQUIRE a newline: it makes it feasible to find the start of the next JSON value from anywhere within a sequence (ie resync).

I think that's right, sequence encoders should be REQUIRED to emit a
newline after any JSON text (and possibly other whitespace).  (FTR, jq
does always emit a newline after any JSON text.)

> The draft 01 rule of requiring a whitespace, but not necessarily a newline, is the worst solution. It still allows interop problems if people rely on lenient implementations. It doesn’t guarantee t…

Agreed.  I'll change it.

>   JSON-sequence = *whitespace *(JSON-text *whitespace %x0A *whitespace)

Is '*whitespace %x0A *whitespace' valid, considering that %x0A is in
whitespace?  A greedy parser might choke on that!

Just in case:

  JSON-sequence = *whitespace *(JSON-text *whitespace2 %x0A *whitespace)
  whitespace = %x20 / %x09 / %x0A / %x0D
  whitespace2 = %x20 / %x09 / %x0D
  JSON-text = <given by RFC7159>

I don't have a similar workaround for the resync heuristic ABNF.

> This section would be better if it focussed on being able to detect the start of the next JSON value from anywhere in a JSON sequence, ie resynchronise.

Good point.  It's nice to be able to seek to arbitrary offsets and
find the next entry.  Traditional syslog-style log files allow that.

> This can be useful for random access to a log file, or for recovering after an entry is corrupted (eg truncated due to a power failure part way through a write).

Yes.

> I would drop the details of trying to recover the 1st entry immediately after a truncated entry. I don't think it is feasible in general (eg when values are numbers); you can't even detect some tru…

Clearly truncation of whole entries and truncation of trailing
whitespace can't be detected this way, but that's to be expected and I
don't think it's a problem.  Being able to scan backwards would be
nice though.  Scanning backwards for boundaries that are followed by
texts that parse is a fallible heuristic unless texts have no internal
newlines (so that's a very good reason to forbid internal newlines in
_logfiles_, but probably not as a general rule).

> The 3rd paragraph no longer applies as whitespace (or hopefully newline) is always required.

I don't agree that it's not required.  If you have a json_loads() type
function that wants to be fed a complete text, and if input texts have
internal newlines, then you might have to repeatedly try json_loads(),
first with one line, then two, ...  This is clearly not a good idea,
and that's what that paragraph warns about.

> Typo: "form a logical end" to "from a logical end"

Finding the nth entry might require a linear search for it!  Finding
the first entry from an arbitrary byte offset into the sequence does
not.

(I've been thinking for a while that I'd like to have a SQLite3
virtual table plugin for use with logfiles that assigns rows an
integer rowid that corresponds to the byte offset into the logfile
where the corresponding log entry is found.  This allows table scans,
and it allows for indexing by rowid.  Then one could create an
auxiliary index mapping any date to the rowid of the first log entry
for that day...  This is would be very fast and handy, and it would
make it possible to use SQL for querying logfiles without having to
build/use keyword indexes.  Whereas assigning log entries logical,
sequential rowids would be much slower!)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 9 May 2014 12:31:20 -0500

> I don't have a similar workaround for the resync heuristic ABNF.

Yes I do.  Never mind.  -02 shortly to be submitted.