Subject: [Json] Advice on registering JSON Lines (not JSON) as IANA Media Type

From: "Hlavina, Wratko (NIH/NLM/NCBI) [E]" <[email-masked]>
Date: Wed, 30 Dec 2020 16:58:13 +0000

Hello, Mr. Crockford and Mr. Williams,

I understand you are listed as the authors for the "application/json" and "application/json-seq" IANA Media Types, respectively.
I would like to ask for your advice/help with a related file format, JSON Lines:

https://jsonlines.org

I think there is value in having this format registered as a Standards Tree IANA Media Type.
Per the RFC6838 process, this requires Expert Review and IETF/IESG approval.
Not being a member of those organizations, how can I encourage such registration?

Motivation:

Unfortunately, JSON Lines is not valid JSON (technically) and is different from JSON Text Sequences.
However, JSON Lines is a frequently used file format; for example, it is used by many database products, including Cloud services like AWS Athena, Snowflake, and others.

Since it is not valid JSON, using "application/json" as media type leads to processing failures and mishandling.
Since it uses the newline as separator, without RS Unicode Information Separator Two record separators, "application/json-seq" is not a substitute Media Type, and the ecosystem of tools do not, in general, support JSON Text Sequences format.

In principle, good JSON programming libraries should allow streamed processing of JSON content, both in emitting it and in reading it, but in practice, libraries for JSON tend to require an entire JSON object to be held in memory.
Since HTTP emits one response per request, this implies only a single JSON object per response, if using "application/json" as Media Type; this is problematic for large data.

In my experience, JSON Lines has become a very useful and conventional file format, since it interoperates well with Unix text utilities while remaining highly interoperable with many JSON tools.

Cf.:

  *   RFC6838
  *   https://www.iana.org/assignments/media-types/application/json
  *   https://www.iana.org/assignments/media-types/application/json-seq
  *   https://www.iana.org/assignments/media-types/application/ld+json
  *   https://stackoverflow.com/questions/51690624/json-lines-mime-type
  *   https://github.com/wardi/jsonlines/issues/9

--
Wratko HLAVINA
Sequence Curation, Organization, Enhancements (Technical Program Manager)
NCBI Building 45 Floor 4 Room AS13D-121
Slack: whlavina / Phone: 301-402-9730 / FAX: 301-480-2484 / Calendar: https://bit.ly/2QU2EGB

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 30 Dec 2020 11:02:47 -0800

Anything that is not JSON should not be called JSON. It should have a 
less confusing name.

On 2020-12-30 8:58 AM, Hlavina, Wratko (NIH/NLM/NCBI) [E] wrote:
> I understand you are listed as the authors for the "application/json" 
> and "application/json-seq" IANA Media Types, respectively.
>
> https://jsonlines.org <https://jsonlines.org>
>
> However, JSON Lines is a frequently used file format; for example, it 
> is used by many database products, including Cloud services like AWS 
> Athena, Snowflake, and others.
>
> Since it is not valid JSON, using "application/json" as media type 
> leads to processing failures and mishandling.
>
> Since it uses the newline as separator, without RS Unicode Information 
> Separator Two record separators, "application/json-seq" is not a 
> substitute Media Type, and the ecosystem of tools do not, in general, 
> support JSON Text Sequences format.
>
> In principle, good JSON programming libraries should allow streamed 
> processing of JSON content, both in emitting it and in reading it, but 
> in practice, libraries for JSON tend to require an entire JSON object 
> to be held in memory.
>
> Since HTTP emits one response per request, this implies only a single 
> JSON object per response, if using "application/json" as Media Type; 
> this is problematic for large data.
>
> In my experience, JSON Lines has become a very useful and conventional 
> file format, since it interoperates well with Unix text utilities 
> while remaining highly interoperable with many JSON tools.
>
> Cf.:
>
>   * RFC6838
>     <https://www.iana.org/assignments/media-types/application/json>
>     <https://www.iana.org/assignments/media-types/application/json-seq>
>     <https://www.iana.org/assignments/media-types/application/ld+json>
>     <https://stackoverflow.com/questions/51690624/json-lines-mime-type>
>     <https://github.com/wardi/jsonlines/issues/9>
>
> --
> *Wratko HLAVINA*
>
> Slack: whlavina / Phone: 301-402-9730 / FAX: 301-480-2484 / Calendar: 
> https://bit.ly/2QU2EGB <https://bit.ly/2QU2EGB>
>

---

From: Stefan Hagen <[email-masked]>
Date: Wed, 30 Dec 2020 20:35:45 +0100

> Am 30.12.2020 um 20:03 schrieb Douglas Crockford <[email-masked]>:
> 
> 
> Anything that is not JSON should not be called JSON. It should have a less confusing name.
> 
> 
> 
>> RFC6838
>> https://www.iana.org/assignments/media-types/application/json
>> https://www.iana.org/assignments/media-types/application/json-seq
>> https://www.iana.org/assignments/media-types/application/ld+json
>> https://stackoverflow.com/questions/51690624/json-lines-mime-type
>> https://github.com/wardi/jsonlines/issues/9
>>  
>> --
>> Wratko HLAVINA

Well, this is JSON texts separated by newline characters.
I think the original JSON sequences proposal started exactly like this (with newlines)
this is how I remember our e-mail discussions - and then the not too surprising practical ivory tower like discussion waves injected the long forgotten RS into the picture. 

Reading the RFC again I suggest to not reuse the json-seq media type in this case, as that specification  assumes skipping to RS tokens between JSON texts which these newline separated JSON streams will not offer.

I suggest to rather request a new media type from IANA and would not object having it start with text/json-

Please enjoy all a healthy and wonderfully non-semantic Year version 2021,
Stefan
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: Tim Bray <[email-masked]>
Date: Wed, 30 Dec 2020 11:49:54 -0800

A survey of https://en.wikipedia.org/wiki/JSON_streaming is instructive.
Among other things, it revealed (to me) the existence of json-seq,
https://tools.ietf.org/html/rfc7464

Doug has a point about naming but I've lost enough arguments about what
things should be called that I'll skip that subject.

In any case, to answer the original question, to register a media type you
need to link to a stable specification.  The contents of
https://jsonlines.org probably don’t qualify, so the conventional thing
would be to write an Internet-Draft which AFAICT would be the same as
json-seq only without the leading "ASCII Record Separator (0x1E)" but
retaining the trailing \n.

Given such an Internet-Draft and evidence that the community actually
cares, it's not difficult to register a new media type.

On Wed, Dec 30, 2020 at 11:36 AM Stefan Hagen <[email-masked]> wrote:

> I understand you are listed as the authors for the "application/json" and
> "application/json-seq" IANA Media Types, respectively.
>
> However, JSON Lines is a frequently used file format; for example, it is
> used by many database products, including Cloud services like AWS Athena,
> Snowflake, and others.
>
>
>
> Separator Two record separators, "application/json-seq" is not a substitute
> Media Type, and the ecosystem of tools do not, in general, support JSON
> Text Sequences format.
>
>
>
> processing of JSON content, both in emitting it and in reading it, but in
> practice, libraries for JSON tend to require an entire JSON object to be
> held in memory.
>
> Since HTTP emits one response per request, this implies only a single JSON
> object per response, if using "application/json" as Media Type; this is
> problematic for large data.
>
>
>
> file format, since it interoperates well with Unix text utilities while
> remaining highly interoperable with many JSON tools.
>
>
>
> Cf.:
>
>    - RFC6838
>    - https://www.iana.org/assignments/media-types/application/json
>    - https://www.iana.org/assignments/media-types/application/json-seq
>    - https://www.iana.org/assignments/media-types/application/ld+json
>    - https://stackoverflow.com/questions/51690624/json-lines-mime-type
>    - https://github.com/wardi/jsonlines/issues/9
>
>
>
> --
> *Wratko HLAVINA*
>
> https://bit.ly/2QU2EGB
> this is how I remember our e-mail discussions - and then the not too
> surprising practical ivory tower like discussion waves injected the long
> forgotten RS into the picture.
>
> Reading the RFC again I suggest to not reuse the json-seq media type in
> this case, as that specification  assumes skipping to RS tokens between
> JSON texts which these newline separated JSON streams will not offer.
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 30 Dec 2020 21:25:19 +0100

On 30. Dec 2020, at 20:49, Tim Bray <[email-masked]> wrote:
> 
> In any case, to answer the original question, to register a media type you need to link to a stable specification.  The contents of https://jsonlines.org <https://jsonlines.org/> probably don’t qualify, so the conventional thing would be to write an Internet-Draft which AFAICT would be the same as json-seq only without the leading "ASCII Record Separator (0x1E)" but retaining the trailing \n.

There is a bit more in which the JSON-sequence RFC differs from what would be a JSON-lines specification.

(Author of RFC 8742 here :-), another derivative from the JSON-sequence idea of registering a media type for a sequence.)

JSON sequences are sequences of arbitrary JSON texts, each enveloped in RS/LF.

JSON-lines are a sequence of LF-delimited lines, each of which is interpreted as a JSON text.

So you can’t have a newline in the JSON text that goes into a JSON-line.
(Which is not a big practical problem, as deleting all newlines from a JSON text creates an equivalent JSON text, even if equivalence is not really specified in the JSON specifications.)

We could have gone that way when the design of JSON sequences was still up in the air.
Some of the robustness properties of JSON sequences do come back with disallowing newlines in the JSON texts (which somehow was not something that was being considered very seriously).
As specified, JSON-lines does not have truncation detection; it needs to be changed to make the LF a line terminator, not a delimiter.

A leading RS is a great signature for JSON sequences; JSON-lines has no such thing.

Grüße, Carsten

---

From: "Hlavina, Wratko (NIH/NLM/NCBI) [E]" <[email-masked]>
Date: Wed, 30 Dec 2020 20:34:51 +0000

Thank you for the feedback.

> [Doug:] Anything that is not JSON should not be called JSON. It should have a less confusing name.
I agree the name of the format leads to confusion.
While JSON Lines is not JSON, it contains items of (a constrained subset of) JSON.
That is, each line in JSON Lines, is valid JSON, but not all JSON is valid as a line in JSON Lines.

> [Tim:] Given […] evidence that the community actually cares,

As consumers of JSON-like data, we face some difficulties:

  *   Tools don't work well with a single large JSON object.
  *   Concatenated JSON, is not valid JSON; protocols such as HTTP permit only 1 response per request, so if we emit concatenated JSON objects of any form, some Media Type is needed, and it can't be 'application/json'.
  *   None of the other alternatives (ND-JSON, JSON Lines, JSON-Seq) are valid JSON.
  *   JSON-Seq has a Standard Tree IANA Media Type, but it's not popular and few tools support it.
  *   JSON Lines is (arguably) popular, since it interoperates well with Unix tools (unlike JSON-Seq).

I/we are not involved in creating the JSON Lines format, but we use it, and have an interest in having some IANA Media Type in order to identify this format.
We are trying to use Media Types the way they are intended, without hackery.
For example, we need appropriate media types for use with OpenAPI RESTful interfaces.

The community seems to be using "application/x-ndjson" as the media type, but it's not official and widespread.
While there has been a desire for a more official designation, progress has been lacking.

Out of self-interest, I/we would like to encourage taking this in the right direction, whether directly or by guiding others.
I am happy to discuss personally with interested parties.

> [Tim:] it's not difficult to register a new media type

Unfortunately, the contributors to ND-JSON and JSON Lines look like they could use some help.
I linked to some of the GitHub issues for the relevant projects; I'll try to get them involved here.

I am hoping there is agreement that registering a new Media Type is justified, and that we as a community just need to follow through on the task.
I note there are dozens of other JSON variants (JSON5, HSON, and many others), but from my experience, I'd question if those have enough adoption to merit a Standard Tree IANA Media Type, at this time.

Cf:

  *   https://en.wikipedia.org/wiki/JSON_streaming
  *   https://github.com/ndjson/ndjson-spec/issues/21
  *   https://github.com/ndjson/ndjson-spec/issues/25
  *   https://github.com/wardi/jsonlines/issues/9
  *   https://github.com/wardi/jsonlines/issues/19
  *   https://github.com/jshttp/mime-db/issues/224
  *   https://github.com/openownership/data-standard/issues/48

From: Tim Bray <[email-masked]>
Sent: Wednesday, December 30, 2020 2:50 PM
To: Stefan Hagen <[email-masked]>
Cc: Douglas Crockford <[email-masked]>; Nico Williams <[email-masked]>; JSON WG <[email-masked]>; Hlavina, Wratko (NIH/NLM/NCBI) [E] <[email-masked]>
Subject: Re: [Json] Advice on registering JSON Lines (not JSON) as IANA Media Type

A survey of https://en.wikipedia.org/wiki/JSON_streaming is instructive. Among other things, it revealed (to me) the existence of json-seq, https://tools.ietf.org/html/rfc7464

Doug has a point about naming but I've lost enough arguments about what things should be called that I'll skip that subject.

In any case, to answer the original question, to register a media type you need to link to a stable specification.  The contents of https://jsonlines.org probably don’t qualify, so the conventional thing would be to write an Internet-Draft which AFAICT would be the same as json-seq only without the leading "ASCII Record Separator (0x1E)" but retaining the trailing \n.

Given such an Internet-Draft and evidence that the community actually cares, it's not difficult to register a new media type.

On Wed, Dec 30, 2020 at 11:36 AM Stefan Hagen <[email-masked]<mailto:[email-masked]>> wrote:

Am 30.12.2020 um 20:03 schrieb Douglas Crockford <[email-masked]<mailto:[email-masked]>>:

https://jsonlines.org

Motivation:

Cf.:

  *   RFC6838

_______________________________________________
json mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/json
_______________________________________________
json mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/json