Subject: [Json] Using JSON in log files

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 20 Nov 2013 11:09:33 -0500

The biggest technical blunder in JSON right now is the insistence that
there only be one object in a file. XML suffers from the same blunder, here
is why.

If you have a log file and it is being updated by multiple processes, there
is potential for contention between the updates. The simplest mechanism for
resolving that contention is to make the log append only and use some
operating system primitive that supports an atomic write to the end of the
file.

I do have code that allows me to append to the end of a JSON log without
reading the whole thing into memory (which is important as my logs are
expected to grow to several MB.) What I do is to work back from the end of
the file until I see "]}" and then I replace that with ", {<New entry>} ]}

But this approach is rather brittle and somewhat risky. It also upsets
various schemes designed to avoid wearing out my flash memory that handle
atomic append operations in a special way.

What I would like is an option for use in log files only that would allow a
beginning of array marker to inform the reader that the end of file marker
is an implicit close for that structure.

So instead of my file starting:
{
  "Keys": [{
      "KeyID": "AAAJMA-ECEYAJ-GABJYPA-G6AAVC-GAB4ACI-FAAFGA-BGZ2AL-LAA",
      "SelfCertificates": [{

It would look something like:

[*
  {"Key": [{
      "KeyID": "AAAJMA-ECEYAJ-GABJYPA-G6AAVC-GAB4ACI-FAAFGA-BGZ2AL-LAA",
      "SelfCertificates": [{
   ...}
  {...}
  {...}

This is not an extension I would want to use in an on-the-wire format. But
for log files it is essential. Particularly in my case where the logs are
digitally signed notary records.

I will be adding a similar feature to JSON-B but I would if possible like
to make whatever change I make there be compatible with any solutions that
other people have developed independently.

I know this is out of scope for the docs, but I think that adding this
feature and comments to the text encoding of JSON would be useful recharter
items.

And yes, I am aware that some people would prefer to keep JSON simple and
not meet the requirements of my use cases. My position is that I want to
use a single encoding for all the non ASN.1 parts of my system and that be
as close to JSON as possible. But I am not going to compromise my system it
the specification is not adequate for my needs.

Writing a JSON reader is very easy, I am quite OK with a requirement that
someone has to write their own reader or put a wrapper around an existing
reader to accept my logs.

-- 
Website: http://hallambaker.com/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 20 Nov 2013 18:02:40 +0100

>digitally signed notary records.

I use http://yaml.org/ for such purposes. It is a superset of JSON, in
a way. Converters to JSON, which you would need one way or another, are
readily available. You could also just cheat and say "It's JSON without
the encapsulating []".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Wed, 20 Nov 2013 09:16:32 -0800

Check how XMPP skates around XML’s single-root-element rule.  Me, I’d just
have a log file containing a sequence of JSON texts.

On Wed, Nov 20, 2013 at 9:02 AM, Bjoern Hoehrmann <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Erik Wilde <[email-masked]>
Date: Wed, 20 Nov 2013 09:31:52 -0800

or just use http://www.w3.org/TR/xml/#TextEntities and happily and 
legally append log data to these. good enough for me. cheers, dret.

On 2013-11-20, 9:16 , Tim Bray wrote:
> Check how XMPP skates around XML’s single-root-element rule.  Me, I’d
> just have a log file containing a sequence of JSON texts.
>
>
> On Wed, Nov 20, 2013 at 9:02 AM, Bjoern Hoehrmann <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Björn Höhrmann · mailto:[email-masked]
>     <mailto:[email-masked]> · http://bjoern.hoehrmann.de
>     Am Badedeich 7 · Telefon: +49(0)160/4415681
>     <tel:%2B49%280%29160%2F4415681> · http://www.bjoernsworld.de
>     [email-masked] <mailto:[email-masked]>

-- 
erik wilde | mailto:[email-masked]  -  tel:+1-510-2061079 |
            | UC Berkeley  -  School of Information (ISchool) |
            | http://dret.net/netdret http://twitter.com/dret |

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 20 Nov 2013 12:43:43 -0500

> legally append log data to these. good enough for me. cheers, dret.

I want to move to a single encoding for the whole system.

Moving to XML does not help in that goal.

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 20 Nov 2013 10:22:24 -0800

This is one of canonical use-cases for streaming JSON processing: just use
linefeed between root-level values. Many/most streaming JSON readers
support this, especially Java ones (Jackson definitely does; GSON has
streaming mode, Genson as well; org.json's toy lib should not be used for
anything).
It also makes it easier to read, should you need to check out log files.

Or, if you want to be strictly compliant with specs, use a container JSON
array.
Streaming readers (and generators) should work fine here as well.
There is no need to buffer the whole log stream in buffer either when
reading or writing.

-+ Tatu +-

On Wed, Nov 20, 2013 at 8:09 AM, Phillip Hallam-Baker <[email-masked]>wrote:

> If you have a log file and it is being updated by multiple processes,
> there is potential for contention between the updates. The simplest
> mechanism for resolving that contention is to make the log append only and
> use some operating system primitive that supports an atomic write to the
> end of the file.
>
> What I would like is an option for use in log files only that would allow
> a beginning of array marker to inform the reader that the end of file
> marker is an implicit close for that structure.
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 20 Nov 2013 14:20:03 -0500

> There is no need to buffer the whole log stream in buffer either when

Thanks, I will do that.

Given that this seems to be a common extension, shouldn't we document it as
such?

What other common extensions are in use?

---

From: Nico Williams <[email-masked]>
Date: Wed, 20 Nov 2013 20:21:47 -0600

On Wed, Nov 20, 2013 at 10:09 AM, Phillip Hallam-Baker <[email-masked]> wrote:

+1

If you look at tools like jq, you'll see that it's perfectly natural
to treat a sequence of JSON texts as an array of values, without any
need to first collect them into a larger JSON text consisting of an
array of all those values.

% echo 1 2 3 4 | jq .
1
2
3
4
% echo 1 2 3 4 | jq -s .
[
  1,
  2,
  3,
  4
]
%

This approach, combined with an online parser, makes it trivial to
deal with log-type structures (append-only with atomic appends) where
one appends JSON texts.

Sure, one could also pretend that the log starts with '[', make sure
every entry is a JSON text followed by a ',', and use an online
parser, but this is more fun.

The one caveat is that numbers are not self-delimiting, which means
that JSON texts consisting of a numeric value must be delimited with
whitespace.

Nico
-- 

Nico
--

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 21 Nov 2013 11:20:16 +0100

> have a log file containing a sequence of JSON texts.

True. However, compared to XML, JSON has an additional issue of "value-separator" (comma) that can only appear between members. Perhaps it would be useful to allow for a comma right before the closing brace/bracket.

Lada

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C