Subject: [Json] More on log file formats

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 5 May 2014 20:53:36 -0400

So I needed a log file today. And I would like it in JSON. But as
previously discussed, JSON does not work very well since we want to be
able to write out log file entries as
an append only operation.

I now have two reasons for why the way to write out the log file
entries is as a sequence of JSON objects without additional comas or
separators. Both have to do with the need to resynchronize.

The log file I am writing looks a bit like this:

{.A...}
{.B...}
{.C...}
{.D...}

Now lets imagine we have a million entries and a log file viewer that
is presenting the entries to the user, 40 at a time since thats what
will fit on the screen. I don't think we want to have to parse the
whole log file to present information buried in the middle of the
file.

Alternatively, lest imagine that the machine crashed while writing
entry C. Do we want this to corrupt all the entries that follow making
them unreadable? I certainly don't think so.

Both use cases create a requirement to be able to resynchronize the
reader to find the start of the next entry. This requires that there
be a sequence that cannot possibly occur within a legitimate JSON
object.

Looking for the sequence '}' WS* '{' allows a reader to resynchronize
to a legitimate start of entry marker.

-- 
Website: http://hallambaker.com/

---

From: Nico Williams <[email-masked]>
Date: Mon, 5 May 2014 20:10:30 -0500

That's a good point about resynchronization, but I can't think of a
fail safe sequence other than by using forbidden non-characters!

Your proposed resync sequence doesn't work.  Consider this:

"foo<process dies here>} {<new entry here>

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 5 May 2014 20:12:16 -0500

There is a solution though: write entries encoded compactly (no
newlines) and a newline after each entry.  The resync sequence then is
just the newline character.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 5 May 2014 21:28:45 -0400

> "foo<process dies here>} {<new entry here>

And I just thought the same thing as soon as I sent the mail.

The only way I can see that we could avoid this completely would be to
forbid newlines in strings and then resync on '}' *ws CR *ws '{'

Actually, JSON already requires control characters to be escaped in
strings. So this works fine.