Subject: [Json] JSON Sequence support for log files

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 16:42:07 -0500

Appending to logfiles creates the need to recover from interrupted
writes (i.e., corrupted entries).  Therefore I propose the following
minor change to the JSON Sequence I-D:

a) add ASCII RS to the whitespace rule
b) RECOMMEND that logfile writers write either RS <entry> RS NL or
<entry-without-internal-newlines> NL

(producing entries with no internal newlines is trivial, as discussed
in other posts)

c) describe how to resynchronize by seeking to the next RS or NL when
a JSON text fails to parse, until a text that parses is found.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 7 May 2014 18:24:55 -0400

Why can't we have } NL { as a record boundary separator?

It is consistent with text encoding, does not require use of unusual
characters and is easily detected using a simple FSR:

Start:   ['}'  S1]  [* Start]
S1: [NL S2] [WS S1] [* Start]
S2: [WS S2] ['}' Boundary] [* Start]
Boundary:

No need to constrain the text in records, no need to introduce any
unusual characters or perform unnatural acts.

I read log files with an editor by choice because a text editor can do
anything that a terminal with line mode commands can do but much more
easily. I usually look for particular records to find an event and
then go backwards and forwards from that point.

Just as the only server TCP ports that work any longer are 80 and 443,
the only ASCII control characters that work are LF and TAB. Even CR is
starting a (long overdue) decline. Most program languages accept LF
and CRLF terminations as completely interchangeable.

On Wed, May 7, 2014 at 5:42 PM, Nico Williams <[email-masked]> wrote:
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

-- 
Website: http://hallambaker.com/

---

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 17:49:21 -0500

> Why can't we have } NL { as a record boundary separator?

JSON Sequence is more general than logfiles.  As such it supports any
type at the top-level, therefore } NL { doesn't work.

For logfiles, however, it does work, as we can constrain the type used
at the top-level in logfiles.

In fact, any sequence of ( %x22 / "]" / "}" ) %x0A ( %22 / "{" / "[" )
works!  Thus even for logfiles we can have arrays, objects, and
strings at the top-level.

> then go backwards and forwards from that point.

Ah, you _read_ them with $EDITOR.  Got it.  Still, it works; try it.

> and CRLF terminations as completely interchangeable.

HTTP still uses \r\n.  To my chagrin.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 8 May 2014 01:20:02 +0200

Off-topic (I don’t have an opinion on the on-topic discussion), but somebody was wrong on the Internet:

> the only ASCII control characters that work are LF and TAB

(There is no TAB character in RFC 20.  I think you mean HT.)

If you think that HT “works” for any definition of that word I don’t know where to start.

RFC 20 is exactly right:

   HT Horizontal Tabulation
      (punched card skip)

Don’t use HT outside the domain of punched cards.

(Well, it was useful with the LA36† at 300 bit/s because it advanced the carriage so much faster than eight spaces would.
[I’m too young to have extensive experience with ASR33s.]
But that replacement could be done in the line driver so it never had a place in source files.)

By the way, a quite respectable number of people*) use FF characters as information separators.  FWIW.

Grüße, Carsten

*) Like, us, when we write RFCs.  Does your editor blow up on those, too?

† http://www.columbia.edu/cu/computinghistory/la36.html

---

From: "Manger, James" <[email-masked]>
Date: Thu, 8 May 2014 11:45:01 +1000

> strings at the top-level.

Syncing to a boundary in the middle of a JSON sequence or log isn't a good reason for RS. You can sync without RS. I think the following detects a boundary between any JSON values.

( "}" / "]" / %x22 / "e" / "l" / DIGIT ) *ws NL *ws ( "{" / "[" / %x22 / "t" / "f" / "n" / "-" / DIGIT )

RS adds complexity (eg extra steps to strip RSs before passing to JSON parser).
It also causes unexpected behaviour from tools that handle text. For instance, the Chrome browser usually *displays* text files in the browser window, but if it has an RS character the file is downloaded instead!

RS could conceivably help detect *slightly* more cases of accidentally truncated writes (in the middle of a top-level number, in a top-level string in the middle of an escape sequence). That is not compelling. Particularly as it still fails to detect truncation of entire records (including RSs) so it is useless as a proper integrity mechanism. Finally, the "RS <entry> RS NL" suggestion misuses the normal semantics of a separator by using 2 per entry instead of 1.

-1 to RSs

--
James Manger

---

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 22:20:42 -0500

> Syncing to a boundary in the middle of a JSON sequence or log isn't a good
> reason for RS. You can sync without RS. I think the following detects a
> boundary between any JSON values.
>

Oh, good point.  Of course, it's easier to seek to a single byte, but since
I'd not require RS, might as well accept a somewhat more complex resync
process.

Note that this does require prefixing texts with whitespace.

RS adds complexity (eg extra steps to strip RSs before passing to JSON
> parser).

Both add complexity, but I agree that your approach is better than RS.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Thu, 8 May 2014 00:57:30 -0400

> [I’m too young to have extensive experience with ASR33s.]

They ignored it, hence the TAB3 mode of termios.h.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Here lies the Christian, judge, and poet Peter,
Who broke the laws of God and man and metre.

---

From: Nico Williams <[email-masked]>
Date: Thu, 8 May 2014 17:13:23 -0500

> ( "}" / "]" / %x22 / "e" / "l" / DIGIT ) *ws NL *ws ( "{" / "[" / %x22 / "t" / "f" / "n" / "-" / DIGIT )

Yes, it does, but it's only guaranteed to find boundaries where the
preceding text is complete.  This means that we need an
optional/recommended heuristic for scanning backwards to find any
valid/complete texts between the truncated/invalid one and any
boundaries found with your rule.  Otherwise one might be able to log a
bogus entry so as to hide the subsequent one!

Still, I like it.  I'll be submitting -01 soon.

Nico
--