Subject: [Json] JSON Log file encoding JSON-L

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 5 May 2014 23:39:57 -0400

This encoding is based on JSON but for reasons explained in the draft
is not pure JSON. Instead a JSON log is a sequence of JSON objects
separated by newline characters.

There is clearly a need for a way to use JSON in log files. It would
be better to have one than many ways. So probably the best next steps
would be for people to try it and then go for an independent
submission for a MIME type unless people want to do something more
formal.

This scheme does not address the other major use case raised which is
writing configuration files. Configuration files need comments. I
think that is a totally different use case to log files though.

http://datatracker.ietf.org/doc/draft-hallambaker-jsonl/

The spec itself is pretty short.

3. Specification

   A JSON-L log file consists of a sequence of zero or more JSON objects
   as specified in [RFC7159] separated by white space that includes at
   least one newline character.

           JSON-L-text = *(object x-ws)

           x-ws = *ws %x0A *ws

3.1. Resynchronization

   Since control characters are not permitted inside JSON string values
   and JSON objects MUST be separated by commas inside a JSON array, the
   sequence '}' *ws LF *ws '{' can only occur at the end of one log
   entry and the start of the next:'

           Boundary = end-object x-ws begin-object

Not got round to a security considerations yet.
-- 
Website: http://hallambaker.com/

---

From: Nico Williams <[email-masked]>
Date: Tue, 6 May 2014 02:01:46 -0500

For a variety of reasons I would prefer that a log file format based
on JSON simply forbid the appearance of unescaped newlines in JSON
texts, and otherwise be just JSON sequences:

 - it's compatible with JSON sequences
 - it works with line-oriented Unix tools (e.g., wc(1), grep(1), ...)
 - aesthetically it's nice
 - it's simpler to resync: just search for the newlines (or EOF, or
offset 0) around entries that fail to parse

 - it's easy to implement because most JSON encoders I've seen (all?
I think so) have an option for producing "compact" output,
usually/always meaning no whitespace will be emitted between the
tokens that make up a JSON text -- just use this feature and that's
that

 - it even works without compact JSON encoding if you're lucky to not
have crashes and such

   Some OSes don't guarantee that writes to files will be completed
when a process is SIGKILLed.  E.g., recent Linux kernels don't
guarantee that more than one byte will be written in that case -- this
was well-intentioned in that a process could start an enormous write
that an admin might want to stop, but how would they? which is why
Linux now allows SIGKILL to terminate incomplete writes.  Therefore
power failures and system crashes are not the only thing to worry
about.

   Even with non-compact entry encodings, one can always recover with
some heuristics, and at some cost.  The biggest problem here is the
SIGKILL problem.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 6 May 2014 08:05:59 -0400

I am against requiring every entry to be a single line.

The reason for wanting log entries in JSON is to be able to log
structured data. For example a log of updates to a database. Each
update could be several tens of KB long.

Not allowing newline inside an object requires quite a lot of changes.

This does not prevent an application outputting data in a one entry
per line format where that is convenient. And I would expect that to
be the default for many situations. But making it a requirement seems
to assume that the entries are always going to be a screen width long
or less.

---

From: Tim Bray <[email-masked]>
Date: Tue, 6 May 2014 07:20:31 -0700

If you want to use line-oriented tools, don't use JSON.  The no-newlines
rule seems really outside the spirit of JSON.
On May 6, 2014 12:01 AM, "Nico Williams" <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 6 May 2014 10:54:40 -0400

If people want line mode then they probably want something like this:
http://www.w3.org/TR/WD-logfile.html

What I don't like about the W3C log file format is that every entry
has to have every column stated. Which makes it rather inconvenient
for tracking occasional errors. Most cases I just want to log the
source IP and the resource recorded. But if there was something
strange going on then I would probably want the full header set and
more.

If the information is in the logs then people will develop tools to
manage them. Its not like grep and its ilk are fixed and immutable.

One addition that I should probably make is to reference the W3C
specification as one possible source of JSON tags. That way tools can
easily support either format:

So this:

#Version: 1.0
#Date: 12-Jan-1996 00:00:00
#Fields: time cs-method cs-uri
00:34:23 GET /foo/bar.html
12:21:16 GET /foo/bar.html
12:45:52 GET /foo/bar.html
12:57:34 GET /foo/bar.html

Would become:

{ "Version": 1.0,
  "Date": "12-Jan-1996 00:00:00"}

{"time": 00:34:23, " cs-method" : "GET", "cs-uri": "/foo/bar.html}
{"time": 12:21:16, " cs-method" : "GET", "cs-uri": "/foo/bar.html}
{"time": 12:45:52, " cs-method" : "GET", "cs-uri": "/foo/bar.html}
{"time": 12:57:34, " cs-method" : "GET", "cs-uri": "/foo/bar.html}

Fun fact: the W3C log file specifies the time as GMT, not UTC and this
was not a mistake. At the time there was a possibility that GMT would
deviate from UTC by dropping the leap second idiocy and fixing to TAI.
Making corrections on the order of fifteen seconds a century is really
a piffling waste of time when solar time varies by five minutes over
the course of a year.

Of course now that the shutdown of ITU is a medium term possibility,
this might well be revisited.

---

From: John Cowan <[email-masked]>
Date: Tue, 6 May 2014 15:22:03 -0400

> rule seems really outside the spirit of JSON.

I think they fit together quite well, with each line a separate JSON
value.  It's not like it's hard to eliminate whitespace newlines and
escape significant ones (if any) when doing JSON output.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
The present impossibility of giving a scientific explanation is no proof
that there is no scientific explanation. The unexplained is not to be
identified with the unexplainable, and the strange and extraordinary
nature of a fact is not a justification for attributing it to powers
above nature.  --The Catholic Encyclopedia, s.v. "telepathy" (1913)

---

From: John Cowan <[email-masked]>
Date: Tue, 6 May 2014 15:24:24 -0400

> update could be several tens of KB long.

Agreed, but modern GNU or BSD tools can handle pretty much unlimited
line lengths nowadays.

> Not allowing newline inside an object requires quite a lot of changes.

Not really.  For whitespace newlines, it's just a matter of not
generating them.  For newlines in strings, just add them to the
list of characters which must be escaped anyway.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Arise, you prisoners of Windows / Arise, you slaves of Redmond, Wash,
The day and hour soon are coming / When all the IT folks say "Gosh!"
It isn't from a clever lawsuit / That Windowsland will finally fall,
But thousands writing open source code / Like mice who nibble through a wall.
        --The Linux-nationale by Greg Baker

---

From: Nico Williams <[email-masked]>
Date: Tue, 6 May 2014 14:39:01 -0500

> I am against requiring every entry to be a single line.

I didn't say "require".

Given that newlines can only happen outside strings in JSON texts, and
if all entries are surrounded by newlines (effectively; more on this
below), then recovery from incomplete writes is feasible even without
requiring that no newlines appear within a text:

    j = json_loadf(input_file);
    if (json_is_error(j))
        readline(input_file); /* discard to EOL */

That in a loop, of course.  It's not perfect, but if you really want
to not lose entries then you won't O_APPEND them -- O_APPEND's
semantics are too loose.

Every text should be prefixed with a newline.

> update could be several tens of KB long.

Yes of course.  A write could be large enough to tempt a sysadmin to
SIGKILL the writer, leading to truncated entries.

> Not allowing newline inside an object requires quite a lot of changes.

Not at all.  First, you can just apply s/\n//g to the JSON text, thus
not requiring _any_ special encoder features.  Second, Jansson,
json-c, libjv (part of jq), all support "compact" encoding, and I
suspect most encoders do too.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 6 May 2014 14:41:50 -0500

> escape significant ones (if any) when doing JSON output.

Right.  Eliminating internal newlines in JSON texts is just an
s/\n//g!  But I wouldn't require it, just encourage it, at least as
long as we can develop a heuristic for recovering from truncated
writes (see separate reply).  But I'd be OK with requiring it too
since s/\n//g is trivial.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 6 May 2014 22:42:23 +0200

> separated

There is some exciting technology for information separators that is documented in RFC 20, in particular in the penultimate paragraph of its section 5.2.
Transfers right to RFC 3629 without any problems, and works very well with unrestricted JSON texts.

Grüße, Carsten

PS.: Yes, I have briefly worked with Qunix/QNX in the early 1980s.

---

From: Nico Williams <[email-masked]>
Date: Tue, 6 May 2014 16:57:11 -0500

> Transfers right to RFC 3629 without any problems, and works very well with unrestricted JSON texts.

Oh, that's funny.  I like it!

(We've become so accustomed to not having a use for FS/GS/RS/US that
we've forgotten about them.  Or at least I have.)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 6 May 2014 19:15:15 -0500

Thank you Carsten.  I think I'll do the following: a) add RS to the
whitespace list for JSON text sequence separators, b) RECOMMEND
surrounding JSON texts with RS in log-like sequences.  Then JSON
sequences will be meet the needs of logs.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 6 May 2014 20:38:13 -0400

If you use RS it will be impossible to generate these logs in a text editor.

That is a show stopper for me.

---

From: Nico Williams <[email-masked]>
Date: Tue, 6 May 2014 19:46:57 -0500

> If you use RS it will be impossible to generate these logs in a text editor.

Eh?  I can do it in my $EDITOR.  ^V

> That is a show stopper for me.

You... generate logs with $EDITOR?!

Nico
--

---

From: John Cowan <[email-masked]>
Date: Tue, 6 May 2014 21:18:53 -0400

> If you use RS it will be impossible to generate these logs in a

Here's one I generated in "ex", which is a text editor: [].

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
I come from under the hill, and under the hills and over the hills my paths
led. And through the air. I am he that walks unseen.  I am the clue-finder,
the web-cutter, the stinging fly. I was chosen for the lucky number.  --Bilbo

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 6 May 2014 22:05:01 -0400

And how do you know where these control characters are?

If we are going to start using obscure ASCII control characters then
we might as well co all binary. Trying to make use of them is going to
make printers and all sorts of editors barf.

Being able to read them in one or two editors is not enough. If it is
going to be any use then they have to work in 100% of the editors
already out there.

On Tue, May 6, 2014 at 9:18 PM, John Cowan <[email-masked]> wrote:
> Here's one I generated in "ex", which is a text editor: [ ].
>
> --

---

From: "Manger, James" <[email-masked]>
Date: Wed, 7 May 2014 12:05:10 +1000

Don’t do that. Surely no one uses RS anymore; and certainly no one who uses JSON. It doesn't make much sense to use SEPARATOR TWO without using ONE, THREE and FOUR.

[RS = Record Separator = U+001E = INFORMATION SEPARATOR TWO]

We now have Nico’s JSON Text Sequences [draft-williams-json-text-sequence],
and Phillip’s JSON Log Format (JSON-L) [draft-hallambaker-jsonl].

They are basically the same. Can we just keep one? Probably the more generic JSON Text Sequences. 
A JSON-L is a JSON Text Sequences where each item is an object.
If a specific log spec is necessary (eg want to standardise log entry elements like "time") it could say "a JSON-L log is a JSON Text Sequence, where the items MUST be objects and these common elements are defined: time, ...".

--
James Manger

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 6 May 2014 22:16:13 -0400

We do? Ooops.

Didn't see the other draft.

I think the discussion moved on a bit from Nico's. He has a lot of
discussion I think we have solid reasons to make decisions on.

Importing WD-logfile tags seems like a win to me (but I might be biased).

---

From: "Manger, James" <[email-masked]>
Date: Wed, 7 May 2014 12:43:16 +1000

> Importing WD-logfile tags seems like a win to me (but I might be biased).

Maybe, I'm a bit sceptical. Is WD-logfile widely used?
A JSON version of WD-logfile needs more work. For instance, the latter has "directives" and "entries" distinguished by a leading "#". How do you tell if a JSON-L object is a "directive" or "entry"?

P.S. The example in WD-logfile (960323 version, not 960221) and repeated in JSON-L is wrong.
"#Date: 12-Jan-1996 00:00:00" should be "#Date: 1996-01-12 00:00:00".
A JSON version should at least use "1996-01-12T00:00:00Z".

In any case, the syntax & semantics of log entry elements can be separate from the syntax of a resynch-friendly append-friendly series of zero or more JSON values.

--
James Manger

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 6 May 2014 23:01:42 -0400

W3C extended log file format is supported by Apache, IIS and most log
file analysis tools.

Interesting that you mention there is an error in the example, that
was the change made when the document was revised. Seems that the old
version has been substituted for the current somehow when they
reformatted it.

I don't think we need to restrict ourselves to one set of tags or
semantics. And I did think about whether we should update the date
format BTW. But it is useful to have some reference point.

---

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 11:40:06 -0500

> Don’t do that. Surely no one uses RS anymore; and certainly no one who uses JSON. It doesn't make much sense to use SEPARATOR TWO without using ONE, THREE and FOUR.

RS would be used to help distinguish incomplete (because interrupted) entries.

> We now have Nico’s JSON Text Sequences [draft-williams-json-text-sequence],

It's actually a WG work item now, since the charter now includes this work.

> They are basically the same. Can we just keep one? Probably the more generic JSON Text Sequences.

I agree.

However, PHB has a point about recovery from corruption.  My proposed
JSON Sequence work does nothing about that.  An OPTIONAL RS would.

I.e., loggers could (but wouldn't have to) emit RS JSON_TEXT RS NL to
make resynchronization possible.  Other use cases could just emit
JSON_TEXT NL (or any other whitespace.

Nico
--

---

From: Jacob Davies <[email-masked]>
Date: Wed, 7 May 2014 13:12:23 -0700

> And how do you know where these control characters are?

?

> make printers and all sorts of editors barf.

[Citation needed] I tried it in a few places and it either didn't
render at all or rendered as a single space. Since the format requires
sequences of JSON objects, that will result in either "}{" or "} {".

Compared to the alternative, which is requiring that the logged JSON
object have no internal newlines, it seems significantly more readable
to use a character for resynchronization that is already universally
illegal in JSON texts. I suggested RS because that is the actual
intended role of that control character and it is highly unlikely that
any existing JSON emitters pass it unescaped. FF (0x0c) is another
reasonable choice - well, I think you'd want \f\n - but I'm guessing
it's more likely to be erroneously passed unescaped by existing
emitters. Editor support is probably wider, but it's sort of a misuse
of the character.

All that being said I'm less convinced than I was when this was first
discussed that the "no internal newlines or whitespace, one object per
line, use newline for resynchronization" approach is all that bad.
It's not very human-readable but it would work well with existing
tools that operate on lines in log files, and I think most emitters
have a "compact" mode where no extraneous whitespace is included. I
think picking one format is more important than which one it is.

---

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 15:38:21 -0500

>> And how do you know where these control characters are?

With od(1), for example :)

+1 :)

> sequences of JSON objects, that will result in either "}{" or "} {".

Good!

> emitters. Editor support is probably wider, but it's sort of a misuse

The RS thing should be OPTIONAL, to be used when writing entries in a
log-like way (e.g., using O_APPEND), because of the potential for
corruption in the face of interrupted writes.

> think picking one format is more important than which one it is.

Right, writing entries with no internal newlines is easy enough for
log writers, and hardly a problem for anyone consuming logs.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 7 May 2014 16:46:13 -0400

If I open up a file with an RS character in it using notepad, how is
it going to look?

Since RS is effectively an obsolete character the best I would expect
is to see ? all over the place.

Unix and Windows both came to the conclusion that the only control
characters that are used are CR, LF and TAB. And CR is only used
because the Internet adopted CRLF as the line ending sequence rather
than LF.

JSON is a text based encoding. It is not the place to attempt to
resurrect long abandoned control character conventions.

---

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 16:04:48 -0500

> If I open up a file with an RS character in it using notepad, how is

Notepad, Wordpad, my Unix shells, my Unix $EDITOR -- all display it as
a space or not at all.

> is to see ? all over the place.

Nope.  And even so, who cares?  It's a log file, you'll not be editing it.

> because the Internet adopted CRLF as the line ending sequence rather

Try it.

> resurrect long abandoned control character conventions.

Try it.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 7 May 2014 17:18:53 -0400

> Notepad, Wordpad, my Unix shells, my Unix $EDITOR -- all display it as

So how can I look at a file and know if it is a well formed sequence or not?

> Nope.  And even so, who cares?  It's a log file, you'll not be editing it.

I open log files with an editor very frequently.

>> resurrect long abandoned control character conventions.

It is easy enough to detect the sequence '}' *ws* LF *ws '{', its a
four state FSR. Why complicate things with characters that will cause
detours through untested code paths?

---

From: Nico Williams <[email-masked]>
Date: Wed, 7 May 2014 16:40:11 -0500

> So how can I look at a file and know if it is a well formed sequence or not?

Use a JSON text processor, like jq(1).  (I'll have to update jq to
teach it to ignore RS between texts at the top-level.)

> I open log files with an editor very frequently.

First, try it.  Second, what?  Why edit logfiles?  I've been dealing
with logfiles for decades and I've never had to edit one before.  What
are editing them for?

> detours through untested code paths?

It's only necessary to know anything about RS when recovering from a
failure to parse a presumably-truncated entry in the log.  And even
then, we can simply seek forward to the next RS or NL and try parsing
whatever's there.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Wed, 7 May 2014 18:57:21 -0400

> line, use newline for resynchronization" approach is all that bad.

I agree, if by whitespace you mean inessential whitespace.  I don't think
it's reasonable to forbid spaces in JSON strings.

> think picking one format is more important than which one it is.

I'd say just forbidding newlines is sufficient.  There's no reason to
forbid the use of spaces for formatting, or anything else in particular.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Your worships will perhaps be thinking that it is an easy thing
to blow up a dog? [Or] to write a book?
    --Don Quixote, Introduction