Subject: [Json] serializing sequences of JSON values

From: Larry Masinter <[email-masked]>
Date: Sun, 9 Mar 2014 04:27:08 +0000

>  This means the serializer would have to be explicitly told whether to insert space before or after
> writing out data.  

If you want to write out a sequence of JSON values you would write them out as an array with [] around and comma separators.

---

From: John Cowan <[email-masked]>
Date: Sun, 9 Mar 2014 10:36:08 -0400

> If you want to write out a sequence of JSON values you would write them out as an array with [] around and comma separators.

That would be a single JSON value, not a sequence thereof.

-- 
Income tax, if I may be pardoned for saying so,         John Cowan
is a tax on income.  --Lord Macnaghten (1901)           [email-masked]

---

From: "Paul E. Jones" <[email-masked]>
Date: Sun, 09 Mar 2014 12:41:21 -0400

Right. That gets us back to the old definition.  Writing an array or object makes sense, but an integer and then another seems like it just makes things unnecessarily confusing.  Somewhere, somehow there must be whitespace inserted between those.

Paul

---

From: Tim Bray <[email-masked]>
Date: Sun, 9 Mar 2014 09:44:34 -0700

So, all the specifications are now consistent, which is a good thing, but
note the advice in the spec: If you want interoperability, use a list or an
object.  -T

On Sun, Mar 9, 2014 at 9:41 AM, Paul E. Jones <[email-masked]> wrote:

> Right. That gets us back to the old definition. Writing an array or object
> makes sense, but an integer and then another seems like it just makes
> things unnecessarily confusing. Somewhere, somehow there must be whitespace
> inserted between those.
>
> Paul
>
> ------------------------------
> *From:* John Cowan <[email-masked]>
> *Sent:* March 9, 2014 10:36:08 AM EDT
> *To:* Larry Masinter <[email-masked]>
> *Cc:* "Paul E. Jones" <[email-masked]>, "Joe Hildebrand
> (jhildebr)" <[email-masked]>, Tim Bray <[email-masked]>, Pete
> Resnick <[email-masked]>, Bjoern Hoehrmann <[email-masked]>,
> Paul Hoffman <[email-masked]>, "[email-masked]" <[email-masked]>,
> "Matt Miller (mamille2)" <[email-masked]>, Barry Leiba <
> [email-masked]>, "[email-masked]" <[email-masked]>
> *Subject:* Re: [Json] serializing sequences of JSON values
>
> Larry Masinter scripsit:
>

---

From: Tatu Saloranta <[email-masked]>
Date: Sun, 9 Mar 2014 17:33:29 +0000

The way real-life serializers do this is to keep track of state. If a root
value has been output, then insert a separator. If not, then not. Expecting
caller to pass this state would indeed be silly.

I still find it fascinating how much of discussion is based on theoretical
aspect of ifs, whys and maybes. But not in observing how things have been
done for years now.

-+ Tatu +-

On Sun, Mar 9, 2014 at 2:36 PM, John Cowan <[email-masked]> wrote:

> Larry Masinter scripsit:
>
> > If you want to write out a sequence of JSON values you would write them
> out as an array with [] around and comma separators.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Nico Williams <[email-masked]>
Date: Sun, 9 Mar 2014 17:37:00 -0500

Doesn't work for log files...

---

From: "Paul E. Jones" <[email-masked]>
Date: Mon, 10 Mar 2014 05:10:00 +0100

Those I've used keep track of state only insofar as what is required to write out an object or array. At least those I've used do not keep track of state between serialization calls.

In any case, I won't keep belaboring the issue. It's done.

Paul

-------- Original Message --------
From: Tatu Saloranta <[email-masked]>
Sent: March 9, 2014 6:33:29 PM CET
To: John Cowan <[email-masked]>
Cc: Larry Masinter <[email-masked]>, Pete Resnick <[email-masked]>, Bjoern Hoehrmann <[email-masked]>, Paul Hoffman <[email-masked]>, "[email-masked]" <[email-masked]>, "Paul E. Jones" <[email-masked]>, Tim Bray <[email-masked]>, "Joe Hildebrand (jhildebr)" <[email-masked]>, "Matt Miller (mamille2)" <[email-masked]>, Barry Leiba <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Json] serializing sequences of JSON values

-+ Tatu +-

---

From: Jacob Davies <[email-masked]>
Date: Mon, 10 Mar 2014 11:14:29 -0700

Does either the old or new specification say anything about multiple
values in sequence? I'm pretty darn sure the old version at least said
that application/json documents contain one and only one value, and
says nothing about serializing JSON in other formats, so encapsulation
is not addressed. I also don't see how any process that expects to
read sequences of self-delimited JSON objects and arrays is going to
blithely accept numbers and strings and whatnot. I find the idea
unlikely that there is some running code that 1. expects sequences of
JSON values with no delimiters, 2. uses a parser that precisely
followed the JSON specification and rejected anything but JSON objects
and arrays prior to this change, 3. has that parser change under it to
accept all kinds of values and 4. is so indifferent to the contents of
what it is processing that it doesn't notice that it's getting numbers
and strings instead of objects or arrays.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 10 Mar 2014 14:47:43 -0400

On Mon, Mar 10, 2014 at 2:14 PM, Jacob Davies <[email-masked]> wrote:

> blithely accept numbers and strings and whatnot.

True, but there are multiple ways that sequences can be fashioned and this
is a standards organization so we should pick exactly one separator.

The log file use case is critical for me. The stupidity of the restriction
in XML is tedious enough without repeating it in the JSON world. People
want to write append only logs of information. So the sequence syntax does
not work. Having to back up and erase the last closing square brace makes
the logging process slower by an order of magnitude.

At the moment I only log full objects and so there is no ambiguity. But I
would much prefer to have a more general approach. I see the following
options:

A) Observe an implicit delimiter on sequences [...][...] or objects
{...}{...}

B) Observe White space as a delimiter

C) Observe a comma as a delimiter

These are not disjoint sets. I think I am going to change my code to accept
A, B or C. However at the moment I only generate sequences consistent with
A.

I think that C is the approach that fits best with JSON. For better or
worse, JSON uses the comma as an item separator and does not ignore
superfluous entries. So this requires a little more state handling than
absolutely necessary.

My vote would be to require C for the log file use case for consistency but
as I say, I will change my code to accept A, B or C.

I find the idea
> and strings instead of objects or arrays.

I am not sure what 3 or 4 mean. But I have 1 and 2 already. The parser
returns each time that a full object is read. If there is nothing to be
read then it returns a null object. The encoder updates the log using
append only writes that can be flagged as atomic on most O/S.

-- 
Website: http://hallambaker.com/

---

From: "Paul E. Jones" <[email-masked]>
Date: Mon, 10 Mar 2014 19:35:34 +0000

Why would sequences of objects not be preferred for logging.  If I were 
writing to a log file, that would definitely be my preference.  One 
reason is that log files tend to have a number of pieces of information 
to convey per entry, thus they can fit well into a JSON object (even if 
it is a simple object).

Paul

------ Original Message ------
From: "Phillip Hallam-Baker" <[email-masked]>
To: "Jacob Davies" <[email-masked]>
Cc: "Nico Williams" <[email-masked]>; "Barry Leiba" 
<[email-masked]>; "Pete Resnick" <[email-masked]>; 
"Bjoern Hoehrmann" <[email-masked]>; "Paul Hoffman" 
<[email-masked]>; "[email-masked]" <[email-masked]>; "Paul E. 
Jones" <[email-masked]>; "Tim Bray" <[email-masked]>; "Joe 
Hildebrand (jhildebr)" <[email-masked]>; "Matt Miller (mamille2)" 
<[email-masked]>; "Larry Masinter" <[email-masked]>; 
"[email-masked]" <[email-masked]>
Sent: 3/10/2014 2:47:43 PM
Subject: Re: [Json] serializing sequences of JSON values

>True, but there are multiple ways that sequences can be fashioned and 
>this is a standards organization so we should pick exactly one 
>separator.
>
>The log file use case is critical for me. The stupidity of the 
>restriction in XML is tedious enough without repeating it in the JSON 
>world. People want to write append only logs of information. So the 
>sequence syntax does not work. Having to back up and erase the last 
>closing square brace makes the logging process slower by an order of 
>magnitude.
>
>
>At the moment I only log full objects and so there is no ambiguity. But 
>I would much prefer to have a more general approach. I see the 
>following options:
>
>These are not disjoint sets. I think I am going to change my code to 
>accept A, B or C. However at the moment I only generate sequences 
>consistent with A.
>
>My vote would be to require C for the log file use case for consistency 
>but as I say, I will change my code to accept A, B or C.
>
>
>>I find the idea

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 10 Mar 2014 16:09:53 -0400

On Mon, Mar 10, 2014 at 3:35 PM, Paul E. Jones <[email-masked]>wrote:

> writing to a log file, that would definitely be my preference.  One reason
> is that log files tend to have a number of pieces of information to convey
> per entry, thus they can fit well into a JSON object (even if it is a
> simple object).
>

I can't see why I would log sequences of anything other than sequences of
objects. But the choice here is between one of the following forms:

A) [ {..}, {..}, {..} ]
B) {..}, {..}, {..}
C) {..} {..} {..}

Form A is not acceptable as it makes adding data to an append-only log much
more complex. We can't just rely on the O/S atomic append operation. We
have to manage locks etc. That is very very expensive when more than one
process is allowed to write.

Forms B and C are equally acceptable but C looks more like a legal JSON
value.

To parse B I have to write a separate parser, to parse C I use my existing
JSON parser but start it in a different production. Instead of the top
level production being 'object', the top level production is
'array-entries'.

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Mar 2014 20:19:23 +0000

[ Cutting off the silly Cc list... ]

[ No hat ]

On Mar 10, 2014, at 7:35 PM, Paul E. Jones <[email-masked]> wrote:

> Why would sequences of objects not be preferred for logging.  

Because the log is always open for appending, so the sequence never terminates.

This, and many other related topics, were discussed at length earlier on the mailing list and the rough consensus was that the current wording was sufficient.

--Paul Hoffman

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 10 Mar 2014 16:39:26 -0400

On Mon, Mar 10, 2014 at 4:19 PM, Paul Hoffman <[email-masked]> wrote:

> This, and many other related topics, were discussed at length earlier on
> the mailing list and the rough consensus was that the current wording was
> sufficient.
>

But the current wording does not support our use case.

I suggest changing the grammar for array as follows,

OLD:

array = begin-array [ value *( value-separator value ) ] end-array

New:

array = begin-array [ array-values ] end-array

array-values = value *( value-separator value )

This makes it easy for a specification that requires a log file to specify
that the contents of the log are a JSON array-values production. A message
oriented protocol would specify that the message contained a value or an
object production as desired.

---

From: Marc Lehmann <[email-masked]>
Date: Mon, 10 Mar 2014 23:59:01 +0100

On Mon, Mar 10, 2014 at 11:14:29AM -0700, Jacob Davies <[email-masked]> wrote:
> is not addressed.

Keep in mind that a) JSON existed before it was an RFC and b) that the RFC
says a json document contains only one value is unrelated to this issue,
as one should be able to rely on the format specification.

As JSON is (or was...) self-delimited, and this is a common concept among
serialisation formats, it shouldn't be surprising that some people relied
on it.

> I also don't see how any process that expects to read sequences of
> self-delimited JSON objects and arrays is going to blithely accept
> numbers and strings and whatnot. I find the idea unlikely that there
> is some running code that 1. expects sequences of JSON values with
> no delimiters, 2. uses a parser that precisely followed the JSON
> specification and rejected anything but JSON objects and arrays prior to
> this change, 3. has that parser change under it to accept all kinds of
> values and 4. is so indifferent to the contents of what it is processing
> that it doesn't notice that it's getting numbers and strings instead of
> objects or arrays.

Well, the code exists (making this unlikely idea a certainty), I have
written it, and it is used, and it is in trouble now. I also don't
think it is rare, because this situation arises quite naturally in
non-monolithic designs:

   1. you have some application that exchange json texts, using
   2. a protocol that tunnels json texts, which uses
   3. some generic json coder.

neither of these might know about the other, e.g. because the json-tunnel (2
above) is used to connect applications (1 above) on different hosts for
example.

after upgrading the json coder, apps can silently exchange json texts,
and the protocol that worked fine before might silently corrupt data. and
since the json coder is often a generic component that comes with some
"framework" (likewise 2), this situation is quite likely in practise.

Even if unlikely, given the implications, the RFC should not just have
ignored this issue.

As an implementer (with JSON::XS, which so precisely followed the spec
that it found at least one bug in the official testsuite btw.), I am now
in a fairly bad situation - silently switch to the new JSON format and
push the problem to the users, or force them to enable the new version
manually. JSON::XS did support the "new JSON" format via a switch as an
extension, but applications had to explicitly ask for it, so presumably
know what they are dealing with.

Both solutions (silently switching, or defaulting to "old JSON") obviously
don't appeal to me, and having "new json" and "old json" is such a
horrible concept in itself, especially with JSON, which really shouldn't
be split into two different formats after it has been deployed to such an
extent.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      [email-masked]
      -=====/_/_//_/\_,_/ /_/\_\

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Mar 2014 19:41:38 -0500

> unlikely that there is some running code that 1. expects sequences of

This exists:

http://stedolan.github.io/jq

and existed prior to the new RFC, and it happily consumed (parsed),
processed, and produced (encoded) sequences of top-level values of any
JSON value type.

Code exists that uses jq in that way.  jq happens to always emit a
newline after emitting a JSON text.

> and arrays prior to this change, 3. has that parser change under it to

Every JSON parser I've ever used or read the docs for (probably half a
dozen) has an option to parse things other than arrays/objects at the
top-level.

If the sequenced texts are separated by newlines (or, really, any
whitespace) and the parser can be fed a chunk of octets at a time
until a complete text is read, then it's trivial to build an
application that consumes sequences of texts.  There are other parser
features that can be used to build such an application, but since
numbers must be delimited somehow ("on the right"), the simple rule to
follow is to just emit a newline after any JSON text.

> and strings instead of objects or arrays.

Your imagination fails you:

jq 'if type == "number" then ... elif type == "string" then ... elif
type == "array" then ... else ... end'

You can do the same in Python, JavaScript, Ruby, C, and so on and on.

You might argue that this is silly, but I'd say once more that your
imagination fails you.  Suppose I'm grepping for specific values and I
want to output the paths to them in a stream of texts whose form I
know little about -- being new to some project with lousy docs I might
be trying to understand some data.  Why not?

For me the main issue here is that there's an ambiguity (see separate
thread) that is best resolved by always emitting a newline after each
text in a sequence.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Mar 2014 19:48:35 -0500

> know what they are dealing with.

Hmm, no.  No one is forcing any parser or any JSON-using app to handle
non-array/object JSON text sequences if it didn't before.

The only problem I see is the need to disambiguate some cases (see
separate thread), which IMO is best done by always emitting a newline
after every JSON text.

Every parser I've worked with has an option to parse things other than
arrays/objects at the top-level, and every encoder I've ever seen has
an option for compact output (i.e., no newlines).  Given just that and
a convention of emitting a newline after every text then it's
trivially easy to build an application that unambiguously handles
sequences of JSON texts encoding any value type at the top-level.
(It's a bit trickier if the encoder has no option to emit "compact"
texts, since then one might need a parser that can consume inputs
incrementally.)

Nico
--

---

From: Larry Masinter <[email-masked]>
Date: Thu, 13 Mar 2014 21:23:46 +0000

I wonder if there is any lesson from XMPP--  which wanted sequence-of-XML-values in an indefinite length stream -- that would apply to JSON.
What would XMPP look like if it were redone in JSON instead of XML?

If there a tendency to standardize on JSON as a replacement to XML in protocols, then would providing guidelines for how to do so be in scope?

> This, and many other related topics, were discussed at length earlier on the
> mailing list and the rough consensus was that the current wording was
> sufficient.
>

---

From: Matt Miller <[email-masked]>
Date: Thu, 13 Mar 2014 16:23:56 -0600

On 3/13/14, 3:23 PM, Larry Masinter wrote:
> I wonder if there is any lesson from XMPP--  which wanted
> sequence-of-XML-values in an indefinite length stream -- that would
> apply to JSON. What would XMPP look like if it were redone in JSON
> instead of XML?
> 

/me exchanges JSON chair hat for XMPP enthusiast hat ...

If XMPP didn't exist and we instead did JMPP, it might be a series of
discrete objects.  However, the direction of the later "XMPP over <x>"
adaptations (e.g., [BOSH] and [XMPP-WS]) speak toward wanting to use
an existing transport that provides block-oriented framing around
complete structures.

Dealing with partial XML-like structures in XMPP has ranged from
inconvenient to impractical because the vast majority of existing XML
software has assumed a whole XML document when parsing starts.  There
are few libraries that truly allow one to feed XML in fits and starts
and get timely output without some amount of modification or
preprocessing.  When the XMPP community took efforts to apply XMPP
over other transports, the direction has so far leaned toward the
whole documents that most implementations expect; even though the
preprocessing required is rather small, the ability to re-use existing
implementations without first considering oft-unpublished details was
a very powerful motivator.

> If there a tendency to standardize on JSON as a replacement to XML
> in protocols, then would providing guidelines for how to do so be
> in scope?
> 
> 
>> 

/me continues with no hat whatsoever

I think there is value in a guidelines document similar to BCP 79 --
particularly a "don't use JSON for <y>" part.  I'm somewhat
unconcerned on how that is done -- whether that means I-JSON morphs
into that or it is its own separate thing.

- -- 
- - m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

[BOSH] "XEP-0124: Bidirectional-streams Over Synchronous HTTP (BOSH)"
< http://xmpp.org/extensions/xep-0124.html >
[XMPP-WS] "An XMPP Sub-protocol for WebSocket" <
https://datatracker.ietf.org/doc/draft-ietf-xmpp-websocket/ >

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 14 Mar 2014 12:02:39 +0900

Hello Matt,

On 2014/03/14 07:23, Matt Miller wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>

This is very valuable feedback.

As far as I understand, the problem for XMPP has at least two parts:
1) The XML document arrives in pieces.
2) Parsing is done in pieces

My understanding is that parsing in pieces is well covered by SAX 
parsers, so that the problems essentially came from 1). But problem 1) 
doesn't apply to JSON text sequences exchanged as MIME types, and it 
doesn't apply to log files as these are rarely read, and when read, that 
could just be done with something like
 > echo ']' > json-array-close.txt
 > cat logfile.json json-array-close.txt | json-parse
(using [pseudo-]shell syntax).

As for delayed exchange of multiple JSON texts, e.g. a (completely 
hypothetical) JMPP (JSON Messaging and Presence Protocol), one would try 
to use a message-oriented transport (e.g. WebSockets) and then not need 
a concept of JSON text sequences at all.

Regards,   Martin.

---

From: Tatu Saloranta <[email-masked]>
Date: Fri, 14 Mar 2014 04:04:03 +0000

On Fri, Mar 14, 2014 at 3:02 AM, "Martin J. Dürst"
<[email-masked]>wrote:

> My understanding is that parsing in pieces is well covered by SAX parsers,
> so that the problems essentially came from 1). But problem 1) doesn't apply
> to JSON text sequences exchanged as MIME types, and it doesn't apply to log
> files as these are rarely read, and when read, that could just be done with
> something like
> hypothetical) JMPP (JSON Messaging and Presence Protocol), one would try to
> use a message-oriented transport (e.g. WebSockets) and then not need a
> concept of JSON text sequences at all.
>
>
Major challenge with sequence of XML documents is really just implementors,
and inertia.
Having written a fully conformat XML parser that optionally supports
"fragment mode" (Woodstox for Java; implements SAX and Stax APIs) I can say
that compared to all other work, supporting multiple root-level documents
is a minor addition. Mode is used/useful for supporting XMPP, as well as
for use for streaming logging, similar to what has been discussed for JSON.
It has to be explicitly enabled (since such usage is non-compliant, having
multiple roots).

There are some minor details (like xml declaration, what to do if it is
repeated) to deal with,
to allow simple concatenation, but all in all it is not a big technical
problem. I mean, compared to writing actual fully compliant parser
(supporting DTD is 80+% of the deal).

-+ Tatu +-

---

From: Nico Williams <[email-masked]>
Date: Thu, 13 Mar 2014 23:21:08 -0500

On Thu, Mar 13, 2014 at 11:04 PM, Tatu Saloranta <[email-masked]> wrote:
> that compared to all other work, supporting multiple root-level documents is
> a minor addition. Mode is used/useful for supporting XMPP, as well as for
> use for streaming logging, similar to what has been discussed for JSON. It
> has to be explicitly enabled (since such usage is non-compliant, having
> multiple roots).
>

By comparison, parsing a JSON text sequence can be done with any JSON
parser, especially when the encoder is nice enough to use "compact"
encoding options and uses newline as the separator.  To be efficient
it's best to have an incremental parser (that is, that can be fed
chunks of input until the end of the text is found).

That's how jq handles JSON text sequences.  It's really quite simple.

Nico
--

---

From: Tatu Saloranta <[email-masked]>
Date: Fri, 14 Mar 2014 05:27:08 +0000

> That's how jq handles JSON text sequences.  It's really quite simple.
Yes, absolutely. This is also one area where otherwise frustrating feature
of JSON -- requirement that linefeeds must be quoted in text values --
helps.

-+ Tatu +-

---

From: Nico Williams <[email-masked]>
Date: Fri, 14 Mar 2014 02:41:09 -0500

On Mar 14, 2014 12:27 AM, "Tatu Saloranta" <[email-masked]> wrote:
> Yes, absolutely. This is also one area where otherwise frustrating
feature of JSON -- requirement that linefeeds must be quoted in text values
-- helps.

Indeed!  That's the best reason to recommended the use of newline for
maximum interop, maybe even require it.

---

From: Nico Williams <[email-masked]>
Date: Fri, 14 Mar 2014 11:37:16 -0500

The most interoperable form of JSON text sequence would be this:
encoders MUST not use newlines in their JSON text encodings and MUST
follow each JSON text with a newline.

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 14 Mar 2014 17:48:12 +0100

> follow each JSON text with a newline.

+1, and if there is any point in writing this up, we should only write up this most interoperable variant.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Fri, 14 Mar 2014 12:04:40 -0500

> +1, and if there is any point in writing this up, we should only write up this most interoperable variant.

Alright, I'll submit an I-D tonight.  As for MIME types, for all those
apps that expose JSON HTTP APIs but which also deal in HTML, using a
client's Accept (and Content-Type) header to distinguish formats...
we'll need a MIME type for JSON text sequences.

Nico
--

---

From: Matt Miller <[email-masked]>
Date: Fri, 14 Mar 2014 11:24:31 -0600

/me still wears no hat ...

On 3/13/14, 9:02 PM, "Martin J. Dürst" wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
>> 
>>> sequence-of-XML-values in an indefinite length stream -- that
>>> would apply to JSON. What would XMPP look like if it were
>>> redone in JSON instead of XML?
>>> 
>> 
>> If XMPP didn't exist and we instead did JMPP, it might be a
>> series of discrete objects.  However, the direction of the later
>> "XMPP over <x>" adaptations (e.g., [BOSH] and [XMPP-WS]) speak
>> toward wanting to use an existing transport that provides
>> block-oriented framing around complete structures.
>> 
>> inconvenient to impractical because the vast majority of existing
>> XML software has assumed a whole XML document when parsing
>> starts.  There are few libraries that truly allow one to feed XML
>> in fits and starts and get timely output without some amount of
>> modification or preprocessing.  When the XMPP community took
>> efforts to apply XMPP over other transports, the direction has so
>> far leaned toward the whole documents that most implementations
>> expect; even though the preprocessing required is rather small,
>> the ability to re-use existing implementations without first
>> considering oft-unpublished details was a very powerful
>> motivator.
> 
> As far as I understand, the problem for XMPP has at least two
> parts: 1) The XML document arrives in pieces. 2) Parsing is done in
> pieces
> 
> parsers, so that the problems essentially came from 1). But problem
> 1) doesn't apply to JSON text sequences exchanged as MIME types,
> and it doesn't apply to log files as these are rarely read, and
> when read, that could just be done with something like
>> echo ']' > json-array-close.txt cat logfile.json
>> json-array-close.txt | json-parse

There exists a number of SAX parsers that will not emit events until
they have consumed a certain amount of input or a complete XML
document, whichever is first.  The most notable that comes to mind is
in .NET, but I've also encountered some implementations in Java that
behave this way.

There are also a number of platforms where a whole-document XML parser
is available "built-in" but a SAX parser requires additional
modules/libraries/etc (or don't even exist).

JSON is it's own thing, but my experience and observation shows to me
it's not that much different than the journey XML followed.

> hypothetical) JMPP (JSON Messaging and Presence Protocol), one
> would try to use a message-oriented transport (e.g. WebSockets) and
> then not need a concept of JSON text sequences at all.
> 

The XMPP community started with things that mimicked the "raw TCP"
model, but moved to better take advantage of the underlying transports
message-framing, and not deal with sequences at all.

Whenever the bi-annual discussion on "Jabber™ 2.0" or "XMPP 2.0" comes
up, the major discussion is about framing, so we don't repeat our
mistakes of the past.  The next incursion could very well focus on
WebSocket for that framing.

At this point, I personally just don't see a lot of value in having a
media type for JSON text sequences.  I wonder how often a
"Content-Type: application/json-sequence" (or whatever is agreed to)
will actually be encountered over the wire.  Others mile may vary, I
suppose.

- -- 
- - m&m

---

From: Tatu Saloranta <[email-masked]>
Date: Fri, 14 Mar 2014 19:30:50 +0000

> follow each JSON text with a newline.

And presumably this would include all linefeeds that JSON specification
supports? (\r, \n and \r\n).

-+ Tatu +-

---

From: Jacob Davies <[email-masked]>
Date: Fri, 14 Mar 2014 15:38:24 -0700

Requiring that the separator be newline and that the JSON be
serialized without embedded newlines is convenient but seems like an
awkward way to handle splittable sequences when there are other
characters that are not permitted to appear in JSON texts that could
be used with unmodified existing JSON emitters. For instance, form
feed (ASCII 12) or record separator (ASCII 30). Yes, newline has the
advantage of letting you use an existing read-line facility, but
having to explain "A JSON sequence is a sequence of JSON texts - but
they're special JSON texts with no newlines! - separated by newlines"
seems pretty awkward.

I can see how it would be convenient in some cases, but I think it's
better for higher-level structures built on lower-level ones not to
make special requirements of the latter. JSON has always allowed
newlines.

---

From: Nico Williams <[email-masked]>
Date: Fri, 14 Mar 2014 17:50:34 -0500

> make special requirements of the latter. JSON has always allowed

If we can assume incremental JSON parsers then there's no need to say
that the separator may not appear in the JSON texts.

However, we probably shouldn't assume incremental parsers.  And,
conveniently, every encoder I've ever seen has an option for emitting
compact texts which includes not emitting newlines.

Awkward it may be, but it's handy in two ways: 1) the dumbest parsers
can handle it with no efficiency penalty, 2) sending compact texts
saves bandwidth.

jq, for example, has an incremental parser, so it can handle texts
that have embedded newlines just fine.  And it has an option to emit
compact texts.  It has no option for alternative text separators on
the encoder side, but it does accept any JSON whitespace as text
separators on the parser side and does not require a separator when
it's not required to disambiguate texts (i.e., it accepts '[][]' as a
sequence of two empty arrays).

I think allowing implementation with the dumbest parsers is a
debatable goal.  If we don't have it, instead requiring incremental
(or better) parsers, then we don't need to forbid newlines in texts,
we can say any whitespace works as a text separator, and we can say
that the separator may be omitted in some cases.

Let's debate that possible goal: do we need to permit implementation
of JSON text sequence parsing with non-incremental JSON parsers?

I don't have a strong opinion on that.  It'd certainly be most
convenient to require at least incremental JSON text parsing
capabilities, and it'd be less awkward.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 15 Mar 2014 00:01:57 +0100

> On 14.03.2014, at 20:30, Tatu Saloranta <[email-masked]> wrote:
> 
>> On Fri, Mar 14, 2014 at 4:37 PM, Nico Williams <[email-masked]> wrote:
> And presumably this would include all linefeeds that JSON specification supports? (\r, \n and \r\n).
> 

I hope not.

\r newlines no longer exist in the real world.
\r\n on the other hand is just a special case of \n, a sane parser can then just key on \n.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Fri, 14 Mar 2014 16:20:51 -0700

I’m strongly negative about requiring people to avoid newlines in their
JSON in order to be able to string chunks of JSON together into a stream.
 JSON is unambiguously and easily parsable.  To talk in abnf, I don’t think
we should be satisfied with anything less than this:

JSON-sequence = JSON-text *(text-separator JSON-text)
text-separator = 1*(  %x20 / %x09 / %x0A / %x0D )

---

From: John Cowan <[email-masked]>
Date: Fri, 14 Mar 2014 19:32:21 -0400

> text-separator = 1*(  %x20 / %x09 / %x0A / %x0D )

Emphatic +1.

-- 
La mayyitan ma qadirun yatabaqqa sarmadi                            John Cowan
Fa idha yaji' al-shudhdhadh fa-l-maut qad yantahi.              [email-masked]
                --Abdullah al-Hazred, Al-`Azif      http://www.ccil.org/~cowan

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 14 Mar 2014 20:32:41 -0400

On Fri, Mar 14, 2014 at 7:20 PM, Tim Bray <[email-masked]> wrote:

> I'm strongly negative about requiring people to avoid newlines in their
>  JSON is unambiguously and easily parsable.  To talk in abnf, I don't think
JSON-sequence = JSON-text *(text-separator JSON-text)
text-separator = 1*(  %x20 / %x09 / %x0A / %x0D / %2C)

---

From: Nico Williams <[email-masked]>
Date: Fri, 14 Mar 2014 19:50:22 -0500

> text-separator = 1*(  %x20 / %x09 / %x0A / %x0D / %2C)

It's much better to say that a separator must follow every text, since
then there's no need to concern ourselves as to ambiguities.  It also
permits parsers to produce values for texts like 'true ' or '1 ' as
soon as the separator is read rather than waiting for a separator (or
EOF) that might take a while to arrive.

  JSON-sequence = *(JSON-text 1*(text-separator))
  text-separator = %x20 / %x09 / %x0A / %x0D / %2C

I see no value in adding a comma (%2C) to the above.

Nico
--

---

From: "Jim Schaad" <[email-masked]>
Date: Fri, 14 Mar 2014 19:34:23 -0700

From: json [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Friday, March 14, 2014 4:02 PM
To: Tatu Saloranta
Cc: Nico Williams; [email-masked]; Matt Miller
Subject: Re: [Json] serializing sequences of JSON values

 

I hope not.

But I have seen \r\r\n on more than one occasion in the real world

Grüße, Carsten

---

From: Tatu Saloranta <[email-masked]>
Date: Sat, 15 Mar 2014 16:34:27 +0000

> \r\n on the other hand is just a special case of \n, a sane parser can

I don't care deeply one way or another, but since I was assuming "of course
those are accepted" (as per XML, or JSON parsers I'm familiar with), this
needs to be specified to improve interoperability.

-+ Tatu +-