Subject: Re: [Json] Using a non-whitespace separator (Re: Working Group Last Call on draft-ietf-json-text-sequence)

From: Nico Williams <[email-masked]>
Date: Sun, 1 Jun 2014 17:35:00 -0500

On Sun, Jun 1, 2014 at 12:08 AM, Tim Bray <[email-masked]> wrote:
> On Sun, May 25, 2014 at 4:05 PM, Nico Williams <[email-masked]> wrote:
>
>> Currently my thinking is that for backwards compatibility reasons I'd
>> want to to make this RECOMMENDED though, not REQUIRED, except for
>> cases where incomplete writes are a potential problem.
>
> No. There should be only one way to do things.

I'm not terribly fond of this.

It'd be easier if we picked a Unicode whitespace character that's not
used in the JSON whitespace rule and must be escaped in strings,
preferably one that terminals and such generally handle as a
whitespace.  Failing that a character that terminals will ignore.

But since the problem we're talking about is only relevant to log-like
writing patterns, I'd prefer to make this optional for writers,
mandatory for readers.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sun, 1 Jun 2014 17:55:31 -0500

As to Unicode encoding... that's a good point.  I'd say that for JSON
text sequences we should require the use of UTF-8.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 1 Jun 2014 16:10:55 -0700

<no hat>

On Jun 1, 2014, at 3:35 PM, Nico Williams <[email-masked]> wrote:

> whitespace.  

What would a whitespace character be "easier" than a character not allowed in Unicode? It seems to me that a character that could not exist in a string and therefore never needs to be escaped is "easier".

> Failing that a character that terminals will ignore.

I suspect most terminals would ignore FEFF. But I don't think that's actually as important as the fact that it doesn't require escaping.

> writing patterns, I'd prefer to make this optional for writers,

Make what optional for writers and mandatory for readers?

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Sun, 1 Jun 2014 18:12:07 -0500

Oh, right, the separator must be a character that must be escaped in
strings.  That greatly limits the range of characters we can choose
from.

---

From: Nico Williams <[email-masked]>
Date: Sun, 1 Jun 2014 18:13:21 -0500

> strings.  That greatly limits the range of characters we can choose

And it has to be a one-byte character (therefore an ASCII character,
and the texts must be encoded in UTF-8).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sun, 1 Jun 2014 18:16:39 -0500

> What would a whitespace character be "easier" than a character not allowed in Unicode? It seems to me that a character that could not exist in a string and therefore never needs to be escaped is "e…

Because I want text-based tools to be able to at least display (and
grep, and...) JSON text sequence contents much as they can JSON text
contents.

> I suspect most terminals would ignore FEFF. But I don't think that's actually as important as the fact that it doesn't require escaping.

Right, it has to be a character that must be escaped, and its encoding
must require a single byte, otherwise we can't recover simply from
incomplete appends.

> Make what optional for writers and mandatory for readers?

I'd like what jq does now to continue to be valid.  I'd like to add an
alternate separator for the log-file case.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 1 Jun 2014 16:17:50 -0700

<no hat>

> and the texts must be encoded in UTF-8).

Why is that?

---

From: Nico Williams <[email-masked]>
Date: Sun, 1 Jun 2014 19:03:18 -0500

>> and the texts must be encoded in UTF-8).

The problem we're talking about is logfiles where applications
"append" to the logfile.  Incomplete writes can result in some
circumstances.  The question then is: how to recover?  In particular:
how to read past an incomplete entry to the next complete entry?

The I-D currently describes a recovery heuristic, but some reviewers
have stated a preference for a stronger, more easily understood
recovery method.  One obvious approach is to separate texts with a
byte or byte sequence that could not normally happen in a JSON text.
A byte is simpler and easier to handle and understand than a byte
sequence: because the latter can be written incompletely for the same
reasons that a JSON text can.

We've considered all of these approaches (not in this order):

0) newline separator
1) #0 + JSON text boundary detection ABNF
2) #1 + removal of newlines from JSON texts (which does not require
re-encoding, FYI)
3) #2 + write something like a text consisting of a single null value
before every text
4) use some other separator (that isn't a JSON whitespace character)
5) #4 with ASCII RS as the separator

Any separator for #4 has to be something that cannot happen in a JSON
text normally, and it has to be something amenable to recovery from
partial writes.  Even partial writes always at least write complete
_bytes_ (if they write any).  Therefore a one-byte separator is always
unambiguous.

In order for a separator to be utterly unambiguous in the face of
partial writes it has to involve a byte that can never occur in a JSON
text.  I suspect there's no such byte that works for UTF-8/16/32.  If
we limit JSON text sequences to UTF-8 then any ASCII character that
must be escaped in strings and is not valid in the encoding of any
values will do.  That's a very small set of characters.  RS (#5) was
objected to.  I don't see what other character can be used that won't
result similar objections.

The more I think about it the more I prefer the options in the I-D,
#0, #1, #2, and #3.  I don't see better alternatives, and I don't
think this is fatal.

Nico
--

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 02 Jun 2014 10:13:15 +0900

>>>> No. There should be only one way to do things.

I agree with this sentence, but I very strongly disagree with the 
proposal of U+FFFE.

> grep, and...) JSON text sequence contents much as they can JSON text

Yes exactly. The Unicode spec says that U+FFFE must never be interpreted 
as an abstract character nor interchanged. While "interpreted as an 
abstract character" leaves quite a bit of room for interpretation 
(sic!), "not interchanged" is very clear. I hope we won't define a 
format that cannot be interchanged.

The reason why Unicode defines things that way is that there may be 
implementations (think editors,...) that use U+FFFE internally, and 
those would be quite confused when receiving it. There will also be 
applications (editors and other tools, including security checkers) that 
will bark on U+FFFE as part of checking for correct UTF-8.

Regards,   Martin.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 4 Jun 2014 07:30:23 +0000

>5) #4 with ASCII RS as the separator

This is the best one on the list.  It's what RS is for.  If you also want
to allow optional inter-item ignorable whitespace (e.g. \n) for better
printability, you get everything you want.  RS is only used for parser
state synchronization.  RS is forbidden in a JSON text by 7159, so it's
unambiguously a recoverable parse error if it is detected in the middle of
an item.  Grep works fine with this format, particularly when combined
with newlines.  Several editors I tried worked fine with it; one even
rendered the RS as a nice RS-surrounded-by-box glyph.  It's certainly
going to be more interoperable with tooling than U+FFFE, for example.

I also agree with Tim that there should be precisely one way of doing item
separation.  Having to do multiple ways is going to make the parsers
slower and more complicated than they have to be, for no technical benefit.

-- 
Joe Hildebrand

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Jun 2014 07:41:08 -0400

+1 on the separator MUST be an ascii character, not a UNICODE sequence.

Most of my projects don't actually use or need Unicode because they
are low level and don't deal with anything the user needs to see. I
use ASCII tags to make it easier to support non-Unicode languages like
C. Having to move to multibyte Unicode sequences for this one purpose
would be a pain.

If not RS then zero.

On Wed, Jun 4, 2014 at 3:30 AM, Joe Hildebrand (jhildebr)
<[email-masked]> wrote:
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: Nico Williams <[email-masked]>
Date: Wed, 4 Jun 2014 11:12:48 -0500

Alright.  Two proposals:

1) Writers MUST precede texts with RS, and SHOULD follow texts with LF.

(RS must precede at least so as to make any complete text parseable
even if the preceding one was written incompletely.  LF is needed so
as to make line-oriented tooling happy.)

2) Writers of logfiles (or any case where incomplete writes are a
problem) MUST do the same as in (1).  Other writers need only emit
(MUST emit) an LF after each text.  Symmetrically, parsers MUST be
able to parse either form.

I'm partial to (2) because it's backwards interoperable with what jq
does today, but the consensus may well be for (1).

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 4 Jun 2014 10:43:07 -0700

<no hat>

> able to parse either form.

(1) seems much cleaner than (2).

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Wed, 4 Jun 2014 10:54:29 -0700

Hah, I hadn’t realized that RS (U+001E, INFORMATION SEPARATOR TWO) was
excluded.   OK, so the abnf for JSON-sequence becomes one of these two:

JSON-sequence = JSON-text *( %1e JSON-text )
JSON-sequence = *( ws %1e JSON-text )

Depending on whether you see the RS as an initiator or a separator.  I
think I very slightly prefer the second.

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Tim Bray <[email-masked]>
Date: Wed, 4 Jun 2014 10:56:43 -0700

And, editing my own abnf, the two alternatives are:

JSON-sequence = JSON-text *( %1e JSON-text )
JSON-sequence = ws *( %1e JSON-text )

I see no utility in saying anything about linefeeds whatsoever.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Jun 2014 14:01:27 -0400

+1

I prefer the 'strict writer, lose reader' approach here. And that is
not my usual stance.

The reason that I think readers need to be tolerant is that they
should be able to read log files after they have been 'damaged' by
tools that strip out the RS characters.

For example, lets say that I have some program that records every
transaction to a logfile and it is discovered that one of the
transactions was wrong and is corrupting the database. The simplest
solution is usually to take the log file, find the broken transaction,
edit it out and rebuild the data base.

Given the quality of editing tools available on many machines, I don't
trust them to preserve non printing ASCII characters. Heck, the editor
on ubuntu can't even start in a root account without writing garbage
to the terminal.

So readers should be tolerant.

On Wed, Jun 4, 2014 at 1:54 PM, Tim Bray <[email-masked]> wrote:
> Depending on whether you see the RS as an initiator or a separator.  I think
> I very slightly prefer the second.
>
>

---

From: Nico Williams <[email-masked]>
Date: Wed, 4 Jun 2014 15:23:51 -0500

>> excluded.   OK, so the abnf for JSON-sequence becomes one of these two:

Someone (Carsten?) proposed it earlier.  RS is, indeed, perfect for this.

>> I very slightly prefer the second.

It has to be an initiator for append-write logfiles, as it then marks
the end of a possibly incompletely-written text.  Otherwise you might
lose the first text following an incompletely-written text.

RS could also follow a text, but I prefer LF for this because it's
friendly to line-oriented text tools.  It's harmless to have a
trailing LF (since the JSON-text ABNF allows extra trailing whitespace
anyways).

> I prefer the 'strict writer, lose reader' approach here. And that is

Is that in reference to my (2) or Tim's point?  Since you top-posted,
and since Tim didn't propose a loose reader, I can't tell :)

> tools that strip out the RS characters.

Yes.  If you see something like:

<text> <text> <text>

it should parse, even though it should have been:

RS<text>RS<text>RS<text>

because, why not parse it?

Thus my (2).

To repeat myself, jq doesn't insist on any separator at parse time,
except where the separator is needed to disambiguate.  E.g., if you
have to texts consisting of numbers, or booleans, or null, then the
parser can't parse them without a separator of some sort.

> edit it out and rebuild the data base.

Or leave it in and just skip past it when you parse.

> So readers should be tolerant.

Right, which is one reason that I want the ABNF for writers to be:

    sequence = RS JSON-text LF

but parsers should be more liberal.

I'll grant that if there's an RS present then one can use any kind of
JSON parser to parse the sequence whereas otherwise only incremental
and streaming JSON parsers can be used.  This is the one reason to
require that RS always be written.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 4 Jun 2014 13:36:19 -0700

I totally don’t get why you want the LF involved.  I can’t see how it would
make any difference, could you explain more?

---

From: Nico Williams <[email-masked]>
Date: Wed, 4 Jun 2014 17:39:56 -0500

> make any difference, could you explain more?

Sure.

Short version:

I'm a fan of ASN.1 and PER and XDR.  But it's true that JSON is a
_textual_ encoding and so I can use $PAGER.  That's nice.  And if I'm
going to use $PAGER, then newlines between texts is... necessary.

Long version:

Let's start with this: jq(1) _today_ deals in JSON text sequences as
inputs and outputs.  You can ask jq(1) to output compact texts (like
pretty much all JSON encoders I know) or pretty-print, even colorize
texts, but jq(1) will always output a newline after every text, so
this:

% jq -C . some-file | $PAGER -R
<page through colorized pretty-printed JSON text(s) in some-file>

and the texts in 'some-file' will not run on: each will end in a newline.

Most JSON encoders I know have a "compact" output option: the whole
text with no extra whitespace.  jq(1)'s encoder does too, and so we
can:

% jq -c -f program.jq input-file > output
% wc -l output
<total number of output texts>

And sure, I could just as easily:

% jq length output
<total number of output texts>

and that's more correct even (since if 'output' came from some other
processor, there might be consecutive newlines, giving an incorrect
count with wc(1)).

But I can do all sorts of things with just text.  Like: use grep(1) as
an optimization to avoid parsing non-matching texts:

% grep -i "error.*some.error.or.other" output | jq -C -f
more-specific-filter.jq | $PAGER -R
<colorized pretty-printed texts>

Sure, it's an optimization, but don't diss optimizations.

PHB mentions $EDITOR.  I mention $PAGER.  And as I said, I'm a fan of
binary encodings.

If jq(1) spoke some binary encoding of JSON none of that would work,
but that's OK: if I had and were using such an encoding (that worked
well with sequences) I'd just not expect text-oriented tools to be any
use, or I could throw a re-encoder in the pipe so I could use
text-oriented tools.

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Thu, 5 Jun 2014 16:25:01 +1000

>> JSON-sequence = *( ws %1e JSON-text )

RS as a JSON sequence prefix or separator was a bad idea when discussed a month ago and still is.

* You cannot (easily) enter an RS in notepad.
* You cannot (easily) enter an RS in vi.
* You cannot see an RS.
* An RS causes Chrome to treat a file as binary data, instead of text.
* Cut-n-paste a JSON value with an invisible RS prefix and the result is NOT JSON, ie it will fail with a JSON parser as RS is not allowed in JSON.
* No one uses RS.
* RS is now labelled INFORMATION SEPARATOR TWO, not RECORD SEPARATOR.
* We aren't using INFORMATION SEPARATOR ONE, THREE or FOUR.
* A newline as a JSON value terminator is sufficient to parse a JSON sequence unambiguously.
* RS doesn't work well with APIs that read text by the line.
* Detecting a newline that separates JSON values is more complex than detecting an RS character, but it is not that complex (eg handful of lines of code).
* An RS prefix detects only slightly more cases of accidentally truncated writes (in the middle of a top-level number, in a top-level string in the middle of an escape sequence) -- not enough to be compelling.
* The awkwardness of RS will mean many implementations will be lenient, but leniency becomes "expected" which leads to interop problems.

"A JSON sequence is the concatenation of zero or more JSON values, where each JSON value is terminated with a newline."

Simple to understand. Simple to write. Simple enough to parse. Simple enough to resync from the middle of a sequence. Almost identical recovery from accidental corruption is possible in almost all the same instances regardless of whether an RS prefix or newline suffix is used.

--
James Manger

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jun 2014 03:59:17 -0500

> RS as a JSON sequence prefix or separator was a bad idea when discussed a

I'm happy to go with the current I-D's LF-based recovery mechanism or
variant of it.  If that's not acceptable then something like RS has to be
it.

A third alternative is to abandon logfiles as a use or accept truncated
writes leading to fatal parse errors from the corrupted point forwards.  A
fourth alternative is to not publish on the Standards track, maybe publish
as Informational or Experimental whatever fewer of us think is best.
 Before we get to any of those alternatives I'd like to try to get
consensus.

Remember, it's rough consensus and running code.  I have running code and
I'm open to changing it, but if some views are mutually exclusive and
therefore some have to be on the rough side of consensus, then so be it.

> * You cannot (easily) enter an RS in notepad.

A reason to make it optional, as I proposed -- only logfile writers should
have to emit it.

> * You cannot (easily) enter an RS in vi.

Meh.  !printf '\x1E'.  Also, same answer as above.

> * You cannot see an RS.

But that's harmless.  Especially if you know to expect to find it there.

> * An RS causes Chrome to treat a file as binary data, instead of text.

That could get fixed (e.g., if Chrome learns about this new MIME type).

> * Cut-n-paste a JSON value with an invisible RS prefix and the result is
> NOT JSON, ie it will fail with a JSON parser as RS is not allowed in JSON.

But you can be careful to not cut the RS.  Or we could make it RS SP, to
make it easier to find where to cut.

> * No one uses RS.

That's not much of an argument :)

> * RS is now labelled INFORMATION SEPARATOR TWO, not RECORD SEPARATOR.

Ditto.  And noted.

> * We aren't using INFORMATION SEPARATOR ONE, THREE or FOUR.

Again.

> * A newline as a JSON value terminator is sufficient to parse a JSON

Except in the presence of incompletely-written entries.

> * RS doesn't work well with APIs that read text by the line.

Do you have any examples of such APIs?  Define "well".

> * Detecting a newline that separates JSON values is more complex than
> detecting an RS character, but it is not that complex (eg handful of lines
> of code).

Maybe, and that is my preferred solution.  I'm with you there.

> * An RS prefix detects only slightly more cases of accidentally truncated
> writes (in the middle of a top-level number, in a top-level string in the
> middle of an escape sequence) -- not enough to be compelling.

There are other cases if we have as a goal to recover starting at the very
next full text following the truncated one.  Preceding and following texts
with LF also helps recovery in the cases you mention.  But it's not enough.
 The current I-D covers some alternatives that do remove all ambiguities at
little cost to either the parser or the encoder, including: removing
internal newlines from texts (cheap operation; no need to parse and
re-encode) and/or preceding texts with "null" LF.

> * The awkwardness of RS will mean many implementations will be lenient,
> but leniency becomes "expected" which leads to interop problems.

Parsers shouldn't require it.  Encoders should emit it if there's a chance
of truncated writes.  How can interop problems arise from this formula?
 What am I missing?

> "A JSON sequence is the concatenation of zero or more JSON values, where
> each JSON value is terminated with a newline."
>
> Simple to understand. Simple to write. Simple enough to parse. Simple
> enough to resync from the middle of a sequence. Almost identical recovery
> from accidental corruption is possible in almost all the same instances
> regardless of whether an RS prefix or newline suffix is used.
>

Yes, and I very much like that, right up until one wants to cater to
logfiles and the truncated write problem.  See alternatives above.  We must
choose one.  Two camps are squared off and I'm in the middle.  One will win
or all may lose.

My preference is to say that logfile writers must remove internal newlines
from the texts they write.  That is by far the simplest fix for write
truncation.   Not all sequences will be logfile-like.  No RS if we go with
that.  I haven't seen any strong arguments against that, in fact  You have
stronger arguments against RS than I have seen against internal newline
removal by logfile writers.  Let's sleep on it and revisit tomorrow,

Nico
--

---

From: Jacob Davies <[email-masked]>
Date: Thu, 5 Jun 2014 09:02:34 -0700

I suggested FF if RS was not acceptable. I made the RS suggestion
because it has the correct semantics for this role. (If you prefer
INFORMATION_SEPARATOR_ONE on the basis of this being UTF-8 - fine.) It
also seemed less likely to be accidentally emitted/transmitted than
FF. (I was also curious how many people would oppose it just for being
déclassé; ASCII ain't that big even before you write off swathes of
it.)

Could be worse, it could have been BEL.

FF is also barred from JSON texts, including in non-significant
whitespace: "whitespace characters are: character tabulation (U+0009),
line feed (U+000A), carriage return (U+000D), and space (U+0020)".
Pagers like "more" do pause on FF, but that behavior may be desirable
when you're dumping large, pretty-printed structures.

The accidental inclusion of the separator when cut & pasting is a
reasonable concern. It applies to FF as well as other control
characters though. As a sample, I looked at the Java code for the
json.org parser, Gson, and a parser I wrote: json.org accepts anything
before U+0020 as "whitespace"; Gson accepts only the four whitespace
characters specified (i.e. from what I could tell would reject FF as
well); I used Character.isWhitespace(char) which accepts RS and FF as
whitespace. So behavior here seems hard to rely on.

Ideally one would choose a printing resync character. But JSON
reserves all printing characters for use in strings. It also reserves
all non-ASCII Unicode characters. The *only* characters that work for
separating arbitrary JSON texts are ASCII control characters (and
excluding those mentioned above in the non-significant whitespace
section). All of them potentially suffer from cut & paste issues
because they are all of course illegal in JSON texts. FF is the only
"traditional" whitespace character in that set (i.e. typically matched
by \s in regular expressions) but as seen above, that does not mean
that it will be accepted by maximally-strict JSON parsers.

LF has many advantages beginning with compatibility with existing
line-oriented tools, at the cost of losing the possibility of
pretty-printing and requiring the removal of insignificant newlines
from input JSON texts (which as Nico says can be done mechanically,
without parsing them, so the difficulty is low). Pretty-printing could
be recovered pretty trivially in a filter or in tools. Also Nico is
the one with experience & code here and he says it works well.

However, if you use LF to separate, it should be required of emitters,
and other insignificant whitespace between or inside values should be
barred for emitters, to maximize the usefulness of existing
line-oriented tools. There should not be multiple versions of this
format with different options. I am guessing this will overwhelmingly
be a log format, and dealing with the possibility of truncation and
bad records is always important for logfiles.

On Thu, Jun 5, 2014 at 1:59 AM, Nico Williams <[email-masked]> wrote:
> as Informational or Experimental whatever fewer of us think is best.  Before
> we get to any of those alternatives I'd like to try to get consensus.
>

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Jun 2014 10:01:05 -0700

<no hat>

As a summary from below: you prefer a more normal character like NL plus the need to escape it in strings, versus an obscure character like RS that requires no escaping. Is that correct?

> * You cannot (easily) enter an RS in vi.

It seems like the purpose of draft-ietf-json-text-sequence is to create a format that can be used for log files and other such files that are constantly added to. If so, then the above complaints are not really relevant, right?

> * An RS causes Chrome to treat a file as binary data, instead of text.

Ditto.

> * Cut-n-paste a JSON value with an invisible RS prefix and the result is NOT JSON, ie it will fail with a JSON parser as RS is not allowed in JSON.

That will be true of anything other than a character that doesn't need to be escaped, right? People asked for RS (or something like it) so that they didn't have to deal with escaping when the value chosen was also in a string. 

> * We aren't using INFORMATION SEPARATOR ONE, THREE or FOUR.

All irrelevant. We are creating a new specification.

> * A newline as a JSON value terminator is sufficient to parse a JSON sequence unambiguously.

Sure. And it also causes the need to have escaping.

> * RS doesn't work well with APIs that read text by the line.

Are there JSON APIs that do that?

> * Detecting a newline that separates JSON values is more complex than detecting an RS character, but it is not that complex (eg handful of lines of code).

The people who asked for RS seemed more concerned about escaping newlines in the JSON being written, not detecting it on the incoming. Do you agree that that is also a concern?

> * An RS prefix detects only slightly more cases of accidentally truncated writes (in the middle of a top-level number, in a top-level string in the middle of an escape sequence) -- not enough to be…

That was not the major motivation, however.

> * The awkwardness of RS will mean many implementations will be lenient, but leniency becomes "expected" which leads to interop problems.

That is a prediction of the future.

> Simple to understand. Simple to write. Simple enough to parse. Simple enough to resync from the middle of a sequence. Almost identical recovery from accidental corruption is possible in almost all…

Sure, but it ignores the issue many people had about escaping.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jun 2014 12:06:54 -0500

> As a summary from below: you prefer a more normal character like NL plus the need to escape it in strings, versus an obscure character like RS that requires no escaping. Is that correct?

Just so we're clear, LF already has to get escaped inJSON strings.
Thus using LF in sequences imposes no new constraints on JSON
encoders.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 5 Jun 2014 10:24:13 -0700

> Thus using LF in sequences imposes no new constraints on JSON

Maybe I'm misunderstanding the earlier messages and the current draft. The draft says:

   To make resynchronization reliable, and work both forwards and
   backwards, the writer MUST first ensure that the JSON text being
   written is valid, and SHOULD apply either (or both) of the following:

   1.  Remove internal newlines (not including escaped newlines in
       strings) from any JSON text being written.

   2.  Prefix any JSON text with a null value and a newline.  The append
       write must still be atomic (one write), and contain both texts.

With the change to something like RS, neither of those would be required, right?

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 5 Jun 2014 11:10:20 -0700

I agree with every point Paul made.  I’m completely convinced that we need
an unambiguous can’t-appear-in-JSON separator/prefix.    Anything smaller
than 1F other than the characters included in JSON would be fine by me.
 How about U+0000?

On Thu, Jun 5, 2014 at 10:01 AM, Paul Hoffman <[email-masked]> wrote:

> <no hat>
>
> As a summary from below: you prefer a more normal character like NL plus
> the need to escape it in strings, versus an obscure character like RS that
> requires no escaping. Is that correct?
>
> It seems like the purpose of draft-ietf-json-text-sequence is to create a
> format that can be used for log files and other such files that are
> constantly added to. If so, then the above complaints are not really
> relevant, right?
>
> That will be true of anything other than a character that doesn't need to
> be escaped, right? People asked for RS (or something like it) so that they
> didn't have to deal with escaping when the value chosen was also in a
> string.
>
> > * No one uses RS.
> The people who asked for RS seemed more concerned about escaping newlines
> in the JSON being written, not detecting it on the incoming. Do you agree
> that that is also a concern?
>
> > * An RS prefix detects only slightly more cases of accidentally
> truncated writes (in the middle of a top-level number, in a top-level
> string in the middle of an escape sequence) -- not enough to be compelling.
>

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jun 2014 13:48:23 -0500

On Thu, Jun 5, 2014 at 1:10 PM, Tim Bray <[email-masked]> wrote:
> than 1F other than the characters included in JSON would be fine by me.  How
> about U+0000?

I don't understand Paul's point about escaping.  Can either of you elaborate?

As for NUL, that has all the problems that RS has, and worse.  I'd
rather use RS.  FF even.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Thu, 5 Jun 2014 19:35:46 -0400

>  How about U+0000?

Are you serious?

/me starts jumping up and down yelling KILL, KILL.

There is no reason why a JSON value should ever contain an explicit LF.
Explicit LFs are used solely for decoration.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Andrew Watt on Microsoft:  Never in the field of human computing has so
much been paid by so many to so few! (pace Winston Churchill)

---

From: John Cowan <[email-masked]>
Date: Thu, 5 Jun 2014 19:16:07 -0400

Paul Hoffman scripsit:

> As a summary from below: you prefer a more normal character like NL
> plus the need to escape it in strings, versus an obscure character
> like RS that requires no escaping. Is that correct?

Both require escaping in strings.  The difference is that LF can appear as
insignificant whitespace, whereas RS cannot.  So to achieve the same thing
as RS achieves (restartable/correctable writes), it's simply necessary never
to spit out an insignificant LF.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
And now here I was, in a country where a right to say how the country should
be governed was restricted to six persons in each thousand of its population.
For the nine hundred and ninety-four to express dissatisfaction with the
regnant system and propose to change it, would have made the whole six
shudder as one man, it would have been so disloyal, so dishonorable, such
putrid black treason.  --Mark Twain's Connecticut Yankee

---

From: John Cowan <[email-masked]>
Date: Thu, 5 Jun 2014 19:39:45 -0400

>        strings) from any JSON text being written.

It would be clearer to say "internal non-significant newlines".

> With the change to something like RS, neither of those would be

Correct, but with other problems around data entry and display.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Linguistics is arguably the most hotly contested property in the academic
realm. It is soaked with the blood of poets, theologians, philosophers,
philologists, psychologists, biologists and neurologists, along with
whatever blood can be got out of grammarians. - Russ Rymer

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jun 2014 18:40:14 -0500

> to spit out an insignificant LF.

Or to remove any LFs.  Since unescaped LFs are insignificant in JSON
texts, it's always OK to remove them without parsing and re-encoding.

Your argument is convincing to me: there's no real difference between
RS and LF this way.

Perhaps someone can explain why it would be bad for log writers to
remove [unescaped] LFs from JSON texts?  I can't tell why it's bad.  I
only know that some object.

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Fri, 6 Jun 2014 11:34:05 +1000

> insignificant whitespace, whereas RS cannot.

Correct.

>  So to achieve the same thing

Incorrect.
It is not *necessary* to omit insignificant LF.
It is just *easier* for parsers if every LF terminates a single JSON value (and is never other insignificant whitespace).

With LF as the terminator but also allowed as insignificant whitespace, <endchar> LF <startchar> unambiguously separates JSON values.
This allows you to restart from anywhere in the middle of a sequence.
This allows you to restart after one value is corrupted or truncated.
If you have had some corruption (so we are already in a niche case) and you really don't want to miss the very next value, you are still ok when values are objects or arrays (as log entries will invariably be). The end of the next (non-corrupted) value is found as normal (<endchar> LF <startchar>), then look back for the matching { or [ that started it.

In my judgement it is worth accepting a little extra complexity for JSON sequence parsers (eg having to look for <endchar> LF <startchar>) to make it much easier for JSON sequence producers (no awkward control characters, can be pretty-printed, can be cut-n-paste, can be manually edited, can cat multiple files together, can wrap long lines...).

--
James Manger