Subject: [Json] Third WG Last Call on draft-ietf-json-text-sequences

From: Paul Hoffman <[email-masked]>
Date: Thu, 18 Sep 2014 10:15:38 -0700

Greetings again. The second WG Last Call turned up some interesting clarifications that might make the document more attractive to some reviewers. This WG Last Call is to determine this.

Please respond on the list whether the technical proposal in the new document could be useful in meeting the use cases and scenarios described in the document. Note that this is *not* a request for comments about whether or not you liked the old proposal better; it is a request about the current document.

The current document:

URL:            http://www.ietf.org/internet-drafts/draft-ietf-json-text-sequence-07.txt
Status:         https://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/
Htmlized:       http://tools.ietf.org/html/draft-ietf-json-text-sequence-07
Diff:           http://www.ietf.org/rfcdiff?url2=draft-ietf-json-text-sequence-07

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 18 Sep 2014 19:50:24 +0200

I believe the technical content of the current document is now useful, probably as useful as this approach can get.

I noticed a couple of opportunities for editorial improvement:

— The reference to ISO/IEC 646:1991 should probably be replaced by a reference to RFC 20.
  There is little reason to reference a somewhat hard to obtain (CHF 88!) external document when we have an RFC about the same subject.
— I think the expansion of JSON in the title does a disservice to this document.  This is not about JavaScript.  (I know we’ll have to negotiate with the RFC editor about adding JSON to the “need not be expanded in title” list.)
— RFC 2119 errata.
— the first ABNF needs to be clearly called out as about the sequence of bytes, while the second ABNF is about a sequence of characters.
— "disambiguate JSON-texts consisting of numbers at the top-level.” is not clear about what is being achieved.  There is no ambiguity about numbers in JSON.  2.3 then clarifies what is meant, but the reader is left stumped at the end of 2.2.
— 2.4 s/terminates/indicates/  (the RS does not terminate the old JSON-text, but starts a new one; it is not consumed by the error detection).
— 2.5 “LF instead of RS” is confusing — I don’t know of a format that consistently uses a double LF between JSON-texts.
— s/are not be/are not/ and a few more places where a fine comb might help.

Gruesse, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 18 Sep 2014 18:18:22 +0000

Overall, I'm very happy with the current draft.  I think it solves the 
problem that we set out to solve, and I think it may actually be useful in 
practice (which is a big step up from where I started on this topic).  I 
have a few relatively minor comments:

Section 2.2:

"In prose: any number of JSON texts, each preceded and followed by one
   or more ASCII RS characters and each followed by a line feed (LF)."

This is left over from a previous version.  Proposed text:

"In prose: any number of JSON texts, each preceded by an record separator 
character (RS) and each followed by a line feed character (LF)."

I agree with Carsten that "disambiguate" isn't a good enough word choice.  
Proposed text:

"Following each JSON-text in the sequence with an LF allows detection of 
truncated JSON-texts consisting of a number at the top-level."

Section 2.3:

"Parsers MUST drop JSON-text sequence elements that may have been 
truncated (see previous sentence), but MAY report such texts (including, 
optionally, the parsed text and/or the original octet string)."

I don't see how to implement this combination of MUST and MAY (I think due 
to the ambiguity of "drop", but I think I see what you're getting at.  
Proposed text:

"Parsers MUST NOT report JSON-text sequence elements consisting of 
top-level numbers that may have been truncated in the same way they would 
a complete JSON-text.  Parsers MAY report such texts as errors (including, 
optionally, the parsed text and/or the original octet string).

Section 5, is "shephered" correct, here?

On 9/18/14, 5:15 PM, "Paul Hoffman" <[email-masked]> wrote:

>Greetings again. The second WG Last Call turned up some interesting 
>clarifications that might make the document more attractive to some 
>reviewers. This WG Last Call is to determine this.
>
>Please respond on the list whether the technical proposal in the new 
>document could be useful in meeting the use cases and scenarios described 
>in the document. Note that this is *not* a request for comments about 
>whether or not you liked the old proposal better; it is a request about 
>the current document.
>
>The current document:
>
>URL:            
>http://www.ietf.org/internet-drafts/draft-ietf-json-text-sequence-07.txt
>Status:         
>https://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/
>Htmlized:       
>http://tools.ietf.org/html/draft-ietf-json-text-sequence-07
>Diff:           
>http://www.ietf.org/rfcdiff?url2=draft-ietf-json-text-sequence-07
>
>--Paul Hoffman
>_______________________________________________
>json mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/json
>

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 18 Sep 2014 18:22:03 +0000

On 9/18/14, 5:50 PM, "Carsten Bormann" <[email-masked]> wrote:

>  There is little reason to reference a somewhat hard to obtain (CHF 88!) 
>external document when we have an RFC about the same subject.

Agree.

>— the first ABNF needs to be clearly called out as about the sequence of 
>bytes, while the second ABNF is about a sequence of characters.

The text in the beginning of section 2 is pretty clear about this.  Do you 
want it moved into the sub-sections?

>— "disambiguate JSON-texts consisting of numbers at the top-level.” is 
>not clear about what is being achieved.  There is no ambiguity about 
>numbers in JSON.  2.3 then clarifies what is meant, but the reader is 
>left stumped at the end of 2.2.

Agree.  I provided some potential text in my other message.

>— 2.4 s/terminates/indicates/  (the RS does not terminate the old 
>JSON-text, but starts a new one; it is not consumed by the error 
>detection).

That's a good change.

>— 2.5 “LF instead of RS” is confusing — I don’t know of a format that 
>consistently uses a double LF between JSON-texts.

Removing section 2.5 would be a net win, I think.  Alternately, we could 
state that parsers might also parse other formats without ceasing to be 
JSON-sequence parsers - if we thought that was an interesting distinction 
to make.  I'm not clear that it's interesting yet, however.

>— s/are not be/are not/ and a few more places where a fine comb might 

I love when we get to the point on a doc where typos are the things that 
we're finding!

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 18 Sep 2014 11:24:02 -0700

<very much no hat>

I now have a use case for this functionality, and think that this document now serves that use case well. It seems technically fine.

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 18 Sep 2014 11:39:27 -0700

I can see people using it, and I don’t see any harm in it.

Suggested edits:

lose this:  “Traditionally the way to do this with JSON is to use a
"streaming"
   parser, but these are neither widely available, widely used, nor easy
   to use.”

In fact I’ve never seen one. The paragraph is shorter, simpler, and
stronger without it.

For similar reasons, lose: “without having to have a streaming
   parser (nor streaming encoder)”

s/JSON-text/JSON text/g  The term as defined in 7159 has no hyphen

Lose this sentence: “The rule for encoders, on the other hand, assumes that
   sequence elements are not truncated.”

I don’t understand it and what (I think) it’s saying is implicit anyhow.
 Adds no value.

“see ISO 646-1991” Huh? Why not just do a proper reference to the current
Unicode standard like 7159 does?

I don’t think section 2.5 adds any value, but I’ve said this before and
it’s still there, so I might be in the rough here.

On Thu, Sep 18, 2014 at 10:15 AM, Paul Hoffman <[email-masked]>
wrote:

> whether or not you liked the old proposal better; it is a request about the
> current document.
>
> The current document:
>
> URL:

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Sep 2014 13:58:40 -0500

> In fact I’ve never seen one. The paragraph is shorter, simpler, and

Such things exist for XML...  You pointed out that this format could be
used with XML as well, and there that sentence would definitely be
appropriate.

(I do want to write a streaming parser for jq, but the idea is that
subsets of a streaming parser's output would be reduced by jq programs
into proper values.  That would reduce jq's need for this format, but
it's non-trivial, and the format would remain useful.)

> s/JSON-text/JSON text/g  The term as defined in 7159 has no hyphen

Ah.

>    sequence elements are not truncated.”

OK.

> Unicode standard like 7159 does?

For the parsing rules ASCII is sufficient, though even that is not a
requirement.  We picked RS because its designation in ASCII was
conceptually convenient, but we don't need to say it came from there.

I used ISO 646-1991 only because there's no bibxml for RFC 20(!).

I'd be happy to downgrade this to an informational reference.

> it’s still there, so I might be in the rough here.

But does it do any harm (maybe!)?  What if I change that section's title
to "Historical note"?

It could cause harm if implementors get pressured into supporting the
old format in their parsers.  I think this unlikely to be a problem
though.

I do think this section adds value because there exist apps that use
that format, and while encoders of this format may not produce the old
one, parsers might get a lot of mileage out of handling the old one too.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Sep 2014 14:10:07 -0500

On Thu, Sep 18, 2014 at 07:50:24PM +0200, Carsten Bormann wrote:
> I believe the technical content of the current document is now useful,
> probably as useful as this approach can get.

Great.

> — The reference to ISO/IEC 646:1991 should probably be replaced by a
> reference to RFC 20.  There is little reason to reference a somewhat
> hard to obtain (CHF 88!) external document when we have an RFC about
> the same subject.

I used it only because there's no bibxml for RFC 20.  Oh, wait!  No,
there is, it's RFC.0020.xml.  Fine, I'll switch back to RFC 20.

We could also make the ASCII reference informational, since we don't
really need ASCII, just the byte value 0x1E and a name for it (RS).  We
do need to point out that 0x1E can't happen in a properly formed
JSON-text, and I do point that out, but there's no need to have more
than a normative reference to RFC7159 for this.  The reference to ASCII
should be informational.

> — I think the expansion of JSON in the title does a disservice to
> this document.  This is not about JavaScript.  (I know we’ll have to
> negotiate with the RFC editor about adding JSON to the “need not be
> expanded in title” list.)

Sure.

> — RFC 2119 errata.

Hmm?

> — the first ABNF needs to be clearly called out as about the sequence
> of bytes, while the second ABNF is about a sequence of characters.

The text in section 2.1 says "octets" and "octet string" in several
places.  Is that not enough?

I don't think the text in section 2.2 has to say anything about
characters: it imports JSON-text from RFC7159, so that should be
implied.

> — "disambiguate JSON-texts consisting of numbers at the top-level.”
> is not clear about what is being achieved.  There is no ambiguity
> about numbers in JSON.  2.3 then clarifies what is meant, but the
> reader is left stumped at the end of 2.2.

I could add an example.  RS 3.1415 RS vs RS 3.14159 LF RS -- the latter
cannot have been truncated, the former may or may not have been (perhaps
only the LF got truncated, perhaps many digits of pi got truncated).

OK.

> — 2.5 “LF instead of RS” is confusing — I don’t know of a format
> that consistently uses a double LF between JSON-texts.

Nothing about double-LF should have been implied.

OK.

Thanks,

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Thu, 18 Sep 2014 12:12:53 -0700

> used with XML as well, and there that sentence would definitely be

​XML is a red herring, and in the current draft it says “Traditionally the
way to do this” which I think is actively misleading.  “Tradition” implies
an accepted body of practice, which there isn’t.  You could say something
like “Hypothetically if good streaming software were widely available it
could be used to do this”.  And that is *really* weak-ass.​

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Sep 2014 14:28:20 -0500

> > used with XML as well, and there that sentence would definitely be

ISTR someone claiming to have a streaming parser when RFC7159 was an
Internet-Draft.  (The context was object name dups.)

> XML is a red herring, and in the current draft it says “Traditionally
> the way to do this” which I think is actively misleading.
> “Tradition” implies an accepted body of practice, which there
> isn’t.  You could say something like “Hypothetically if good
> streaming software were widely available it could be used to do
> this”.  And that is *really* weak-ass.

"Hypothetically" works for me, as does "theoretically".  I don't think
that's too weak: the fact that these are difficult-to-use or
non-existent is one of the main motivators for this format.

It's true that the motivation can remain without reference to how one
would handle large, indeterminate-sized data sets with pure JSON.  I
don't oppose removing this, but what harm results from it?  Is it too
distracting?  (It distracted you, of course.)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Sep 2014 14:30:17 -0500

On Thu, Sep 18, 2014 at 06:22:03PM +0000, Joe Hildebrand (jhildebr) wrote:
> >reference to RFC 20.

I hadn't noticed that bibxml exists for RFC.0020.  I'll switch to RFC 20.

> Agree.  I provided some potential text in my other message.

Thanks, I'll take a look.  I'll also add an example as in my reply to
Carsten.

> to make.  I'm not clear that it's interesting yet, however.

Fair enough, I think I'll just remove it then.

> I love when we get to the point on a doc where typos are the things that 

Agreed!

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Sep 2014 14:32:11 -0500

> have a few relatively minor comments:

Thanks!

> character (RS) and each followed by a line feed character (LF)."

Whoops, thanks.

> truncated JSON-texts consisting of a number at the top-level."

I'll take this.

> optionally, the parsed text and/or the original octet string).

I'll take this.

> Section 5, is "shephered" correct, here?

There's a missing 'd'.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 18 Sep 2014 23:23:33 +0200

> Hmm?

http://www.rfc-editor.org/errata_search.php?rfc=2119&eid=499

(Sorry, I thought I could address frequent RFC writers like you with that shorter reference.)

> places.  Is that not enough?

No.  And the text in the intro to 2 says the first ABNF is about bytes vs. the second being about not being truncated (what?).

You need to put in a 5-inch warning sign, in blinking red, because this is really highly misleading usage of ABNF without that warning sign.  (It is also now common behavior that people will just jump to the meat, ignoring the verbiage around.  So every piece of notation or example needs to be self-describing.)

(Insert ceterum censeo about turgid wooden horse-and-buggy grandiloquent term “octet” here.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Sep 2014 16:38:02 -0500

> (Sorry, I thought I could address frequent RFC writers like you with

No, I'd got it, it's just I had never expected an arratum for RFC2119!
I was incredulous (before checking).  Since I don't use the one missing
term, I'm inclined to leave it as-is (the RFC-Editor can always make
this change if they really want to).

> No.  And the text in the intro to 2 says the first ABNF is about bytes
> vs. the second being about not being truncated (what?).

OLD:

      [...].  The rule for encoders, on the other hand, assumes that
   sequence elements are not truncated.

NEW:
      [...].  The rule for encoders, on the other hand, assumes
   well-formed JSON-texts.

> You need to put in a 5-inch warning sign, in blinking red, because
> this is really highly misleading usage of ABNF without that warning
> sign.  (It is also now common behavior that people will just jump to
> the meat, ignoring the verbiage around.  So every piece of notation or
> example needs to be self-describing.)

RFC5234 says:

   Rules resolve into a string of terminal values, sometimes called
   characters.  In ABNF, a character is merely a non-negative integer.
   In certain contexts, a specific mapping (encoding) of values into a
   character set (such as ASCII) will be specified.

There's nothing misleading.

The word "octet" appears eight times in section 2.1, one of those in the
figure with the ABNF.  How much more blinking red can I make it than
that?!  :)

> grandiloquent term “octet” here.)

I picked it for a reason: it goes well with our habit of using OCTET
STRING in ASN.1-based protocols.

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Fri, 19 Sep 2014 10:32:02 +1000

draft-ietf-json-text-sequences-07 is not useful for its primary purpose. Inserting the byte 0x1E means the format is no longer "text" in the usual sense, which ruins much of the rationale for using JSON in the first place. Not allowing whitespace in an empty sequence makes the format very brittle (too easy to create an invalid value when using typical tools such as web templating languages).

Having two different ABNF rules with exactly the same name <JSON-sequence> is bad. Particularly when the terminal values of the first are integers from 0 to 255 (bytes), and the terminal values for the seconds are integers from 0 to 0x10FFFF (Unicode scalars).

1st preference: Drop RS.

2nd preference: Drop the byte-based ABNF for parsing. Only define a text-based ABNF for valid JSON text sequences. Use prose to mention that even if an item in a sequence is not valid JSON (eg a write was truncated) a parser can (and SHOULD) recover subsequent valid items from the sequence.

3rd preference: Use different names for the byte-based and Unicode-based ABNF rules, eg <JSON-seq-bytes>, <RS-byte>.

Special-casing top-level numeric values is awkward. It means you cannot simply split on RS, then parse items as JSON. Instead, you have to: split on RS; note if an item ends with a LF byte; parse item as JSON; if item is a number and there was no LF throw an error. I doubt most implementations will bother with the extra steps, particularly as it only matters in a niche scenario where an app half-cares about integrity: it cares about corrupted numbers but doesn't care about lost items.

Denoting end of sequence "by convention" sounds like bad security advice. A "convention" sounds like a heuristic that an attacker can exploit. Using the Content-Length HTTP header is a good example of how the end can be denoted externally from the format, but I wouldn't classify it as a "convention". A convention might be writing the JSON value null or {} as the last item.

The "interoperability note" is really a warning to expect non-interoperability. Expect some people to get used to more lenient tools, so their data will fail when later used with tools the implement this spec.

The acknowledgements credit me with "ABNF for resynchronization" but that no longer exists.

The abstract has too little content. How about adding a sentence so it becomes:

  This document describes the JSON text sequence format and associated
  media type, "application/json-seq". A JSON text sequence consists of
  any number of JSON values, each prefixed by a Record Separator (U+001E)
  and ending with a newline character (U+000A).

--
James Manger

-----Original Message-----
From: json [mailto:[email-masked]] On Behalf Of Paul Hoffman
Sent: Friday, 19 September 2014 3:16 AM
To: IETF JSON WG
Subject: [Json] Third WG Last Call on draft-ietf-json-text-sequences

The current document:

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Thu, 18 Sep 2014 22:40:42 -0400

> Unicode standard like 7159 does?

Unicode and ISO 10646 don't actually standardize the C0 and C1 control
characters beyond TAB, CR, and LF; it defers to other character set
standards for them.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Fundamental thinking is ha-ard.  Let's go ideology-shopping.
                        --Philosopher Barbie

---

From: Stefan Drees <[email-masked]>
Date: Fri, 19 Sep 2014 07:48:57 +0200

>    and ending with a newline character (U+000A).

+1 from my side on the above as I am also asking myself, if the 
introduction of RS really leaves the approach useful as text data 
transport container standardization.

/Stefan (in a rush, sorry)

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 19 Sep 2014 16:52:40 +0000

>only the LF got truncated, perhaps many digits of pi got truncated).

That's a great idea.  There ought to be an example of a stream of objects 
early in the document as well, so that people can picture what we're 
trying to accomplish.

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 19 Sep 2014 17:34:38 +0000

On 9/19/14, 12:32 AM, "Manger, James" <[email-masked]> 
wrote:

>draft-ietf-json-text-sequences-07 is not useful for its primary purpose. 
>Inserting the byte 0x1E means the format is no longer "text" in the usual 
>sense, 

Could you say more about that, please?  RS has been defined in ASCII since 
roughly the Unix epoch.  There are none of the normal binary issues 
dealing with %x00.  I'm not sure how I could write a definition of "text" 
that didn't include RS.

>which ruins much of the rationale for using JSON in the first place. Not 
>allowing whitespace in an empty sequence makes the format very brittle 
>(too easy to create an invalid value when using typical tools such as web 
>templating languages).

Can you give an example of this, please?  This is a point that I'm more 
sympathetic to.  Also, could you describe why the consequences of an 
invalid value would be worse than I'm imagining?  Most of the reasons I 
can think of to use this format will likely ignore truncation errors.  
Even if this style error is prevalent (which I don't think will be the 
case), and I *did* care about truncation errors, special-casing for 
whitespace seems pretty easy.

>Having two different ABNF rules with exactly the same name 
><JSON-sequence> is bad. Particularly when the terminal values of the 
>first are integers from 0 to 255 (bytes), and the terminal values for the 
>seconds are integers from 0 to 0x10FFFF (Unicode scalars).

It seems like a reasonable way to describe the intent.  To Carsten's 
point, we could add more flashing warning lights, but I think that 
implementers will get it pretty readily.

>1st preference: Drop RS.

At this point, I think that means we would effectively be declaring defeat 
on the document and agreeing not to publish it.  That doesn't seem to be a 
productive approach.

>2nd preference: Drop the byte-based ABNF for parsing. Only define a 
>text-based ABNF for valid JSON text sequences. Use prose to mention that 
>even if an item in a sequence is not valid JSON (eg a write was 
>truncated) a parser can (and SHOULD) recover subsequent valid items from 
>the sequence.

I don't think that is anywhere near as clear for implementors as what we 
currently have.

>3rd preference: Use different names for the byte-based and Unicode-based 
>ABNF rules, eg <JSON-seq-bytes>, <RS-byte>.

This approach doesn't seem unreasonable to me.  I'd suggest 
"JSON-sequence-parse" and "JSON-sequence-generate".

>Special-casing top-level numeric values is awkward. It means you cannot 
>simply split on RS, then parse items as JSON. Instead, you have to: split 
>on RS; note if an item ends with a LF byte; parse item as JSON; if item 
>is a number and there was no LF throw an error. I doubt most 
>implementations will bother with the extra steps, particularly as it only 
>matters in a niche scenario where an app half-cares about integrity: it 
>cares about corrupted numbers but doesn't care about lost items.

I would continue to be ok with just saying "don't send bare numbers", 
because I agree that many implementations will skip this bit.  However, 
I'm fine with what we have now as a compromise.

>Denoting end of sequence "by convention" sounds like bad security advice. 
>A "convention" sounds like a heuristic that an attacker can exploit. 
>Using the Content-Length HTTP header is a good example of how the end can 
>be denoted externally from the format, but I wouldn't classify it as a 
>"convention". A convention might be writing the JSON value null or {} as 
>the last item.

When writing a new entry, you would then have to back up a couple of 
bytes, overwrite the terminator, and write a new one at the end in order 
for the file to be valid.  That defeats the purpose of the format; if you 
want that, just use a plain JSON file surrounded by [].

I don't actually think this problem is bad enough to warrant calling out 
in the security section anyway, but I think one of the ADs wanted it.  I 
don't think the current text harms anything.

>The "interoperability note" is really a warning to expect 
>non-interoperability. Expect some people to get used to more lenient 
>tools, so their data will fail when later used with tools the implement 
>this spec.

We could lose this entire section as far as I'm concerned.

>  and ending with a newline character (U+000A).

I like it.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Fri, 19 Sep 2014 13:18:34 -0500

On Fri, Sep 19, 2014 at 10:32:02AM +1000, Manger, James wrote:
> draft-ietf-json-text-sequences-07 is not useful for its primary
> purpose. Inserting the byte 0x1E means the format is no longer "text"
> in the usual sense, which ruins much of the rationale for using JSON
> in the first place. Not allowing whitespace in an empty sequence makes
> the format very brittle (too easy to create an invalid value when
> using typical tools such as web templating languages).

I've tried using RS with various tools that I use every day, and I had
no problems.  Converting between this format and that which jq supports
today is trivial: remove the RS bytes (assuming no truncated texts).

The primary value in using RS is:

 - no need to remove internal LFs to make sequence parsing and recovery
   easy with any off-the-shelf JSON parser

Note that the format I originally proposed was difficult to implement
with a non-incremental JSON parser.  (jq's parser is incremental but not
streaming, by which I mean that it consumes more octets until a complete
value falls out, and it does not discard octets that start a subsequent
text.)

Using RS and LF resolves all ambiguity issues and makes it trivial to
use even those dumbest (no offense) JSON parsers that want a complete
JSON text as input and nothing more.

> 1st preference: Drop RS.

That was my preference a while back too, as you know.  I've come around
to liking RS.

> text-based ABNF for valid JSON text sequences. Use prose to mention
> that even if an item in a sequence is not valid JSON (eg a write was
> truncated) a parser can (and SHOULD) recover subsequent valid items
> from the sequence.

That would be OK, but would that really improve things?  Anyways, if the
WG agrees...

> 3rd preference: Use different names for the byte-based and
> Unicode-based ABNF rules, eg <JSON-seq-bytes>, <RS-byte>.

OK.

> Special-casing top-level numeric values is awkward. It means you
> cannot simply split on RS, then parse items as JSON. Instead, you have
> to: split on RS; note if an item ends with a LF byte; parse item as
> JSON; if item is a number and there was no LF throw an error. I doubt
> most implementations will bother with the extra steps, particularly as
> it only matters in a niche scenario where an app half-cares about
> integrity: it cares about corrupted numbers but doesn't care about
> lost items.

This only matters in cases where truncation of individual texts is
possible.  If no truncation is possible then removing RSes is enough to
get you a pure stream of unambiguous JSON texts.

> Denoting end of sequence "by convention" sounds like bad security
> advice. A "convention" sounds like a heuristic that an attacker can
> exploit. Using the Content-Length HTTP header is a good example of how
> the end can be denoted externally from the format, but I wouldn't
> classify it as a "convention". A convention might be writing the JSON
> value null or {} as the last item.

The security advice is to use a secure transport if one needs integrity
protection (which one should).  IMO this protocol should do nothing to
denote EOF.  Using a valid JSON text as EOF would be bad!  If an end of
sequence designator were needed I'd propose using NUL or FS (file
separator).

> tools, so their data will fail when later used with tools the
> implement this spec.

Nobody seems to like that section; it's headed for the chopping block.

> The acknowledgements credit me with "ABNF for resynchronization" but

Ah right, thanks for noting that.

>   and ending with a newline character (U+000A).

I like it.  Thanks!

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 19 Sep 2014 13:21:35 -0500

On Fri, Sep 19, 2014 at 07:48:57AM +0200, Stefan Drees wrote:
> > [...]

Near as I can tell the RS doesn't hurt.  What is most needed for a
text-like format is LF, and we now have consensus (right?) for RS
JSON-text LF, which gives us the LF we need to be line-oriented-
tool-friendly, and the RS that we need to fully disambiguate all
cases *and* be friendly to all even the least convenient JSON parsers.

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Mon, 22 Sep 2014 14:12:10 +1000

Joe,

>>Inserting the byte 0x1E means the format is no longer "text" in the 
>>usual sense,

> Could you say more about that, please?  RS has been defined in ASCII since roughly the Unix epoch.  There are none of the normal binary issues dealing with %x00.  I'm not sure how I could write a definition of "text" 
that didn't include RS.

A software library can handle RS, but people will have problems. There is no RS key on my keyboard. I cannot see a RS. Am I including an RS when I cut 'n paste a JSON value? It is displayed (or not displayed) inconsistently by text apps: Notepad shows it like a space; OneNote shows it like a hyphen; BabelMap "shows" it like a zero-width space. It doesn’t behave like the usual text I handle. I wouldn't be surprised if RS is sometimes turned into U+241E SYMBOL FOR RECORD SEPARATOR ␞.

>>which ruins much of the rationale for using JSON in the first place. 
>>Not allowing whitespace in an empty sequence makes the format very 
>>brittle (too easy to create an invalid value when using typical tools 
>>such as web templating languages).

> Can you give an example of this, please?  This is a point that I'm more sympathetic to.

Lots of web pages have a bunch of blank lines near the top and scattered elsewhere because they are generated by templates. "instructions" in the template are omitted from the output, but the text content around the instructions (including the newlines and indentation) do appear. Some version of Java Servlets actually added a trimDirectiveWhitespaces feature to help with this issue (http://stackoverflow.com/questions/208736/strip-whitespace-from-jsp-output), which reflects the problem.

> Also, could you describe why the consequences of an invalid value would be worse than I'm imagining?  Most of the reasons I can think of to use this format will likely ignore truncation errors.  
Even if this style error is prevalent (which I don't think will be the case), and I *did* care about truncation errors, special-casing for whitespace seems pretty easy.

I would prefer not to have to plan for hacks already.

> At this point, I think that means we would effectively be declaring defeat on the document and agreeing not to publish it.

Unfortunate, but better than using RS.

…

>>simply split on RS, then parse items as JSON. Instead, you have to: 
>>split on RS; note if an item ends with a LF byte; parse item as JSON; 
>>if item is a number and there was no LF throw an error. I doubt most 
>>implementations will bother with the extra steps, particularly as it 
>>only matters in a niche scenario where an app half-cares about 
>>integrity: it cares about corrupted numbers but doesn't care about lost items.

>I would continue to be ok with just saying "don't send bare numbers", because I agree that many implementations will skip this bit.  However, I'm fine with what we have now as a compromise.

Saying "don't send bare numbers" would be another special case

…

> We could lose this entire section as far as I'm concerned.

Removing the text doesn't remove the risk of non-interoperability.

--
James Manger

---

From: Stefan Drees <[email-masked]>
Date: Mon, 22 Sep 2014 06:47:59 +0200

On 2014-09-19 20:21 +02:00, Nico Williams wrote:
> ...

I *hope* we have at least consensus for keeping LF in eg. logs ;-)
As consenting adults we and our tools talking with each other on our 
behalf may inject funny things in the info stream if it helps avoid 
misunderstanding. I remember the thrill I still sense when spotting a 
less-than character not being used as an operator but as a 
delimiter/start-of-token in some legacy data format ...

Seriously, I do not fear RS (neither do tools like grep), *but* it makes 
very funny (non-)transforms on the commandline (results of cat or grep), 
being reduced to zero width apearance thus glueing visually together 
what should have been separated by the extra info (represented by RS) in 
the first place.

*But* (horizontal) TAB characters also are similar in these distractions 
they (may) cause when one is just visually inspecting lines including 
them on a shell, but when in doubt there is often a dump utility or 
another unix tool within reach ... in python we even have a nanny ... 
and life is easy again, so:

After some tinking about it, I am now +0 to the draft (including the RS) 
when we all continue chanting "us" as a group into this brave new world 
... and hope remains, that everyone and her dog will now really inject 
RS into there JSON text sequences formerly only split by LFs.

"Stefan"
"Drees"

PS: Note the RS in the signature ;-)
$> $ xxd signature_json_text_sequence.json
0000000: 1e22 5374 6566 616e 220a 1e22 4472 6565  ."Stefan".."Dree
0000010: 7322 0a                                  s".

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 22 Sep 2014 15:14:50 +0000

On 9/22/14, 4:12 AM, "Manger, James" <[email-masked]> 
wrote:

>A software library can handle RS, but people will have problems. There is 
>no RS key on my keyboard. 

You typing JSON sequences is not a use case that I think is terribly 
interesting.  If you know what the contents of the entire file are before 
you start, just use a JSON array.  Note, I'm open to other ways of looking 
at this that I haven't thought of.

>I cannot see a RS. 

Particularly if there is also a LF, I don't see how that matters.  It 
might even be a feature.

>Am I including an RS when I cut 'n paste a JSON value? 

That's a good question.  It's likely inconsistent across tools.  But I 
think any character we pick that has the properties of RS we wanted are 
likely to cause an issue here.

>It is displayed (or not displayed) inconsistently by text apps: Notepad 
>shows it like a space; OneNote shows it like a hyphen; BabelMap "shows" 
>it like a zero-width space. 

Nod.  Again, I don't think there's a better choice.  If you liked, we 
could try to propose how it should be rendered, but I would expect to be 
a) lambasted by the IESG for trying to mandate UI and b) ignored by all 
tool producers.

>It doesn’t behave like the usual text I handle. I wouldn't be surprised 
>if RS is sometimes turned into U+241E SYMBOL FOR RECORD SEPARATOR ␞.

I have a couple of editors which render it as ␞, but preserve it's 
identity for copy/paste - which seems like a perfectly reasonable choice.

>>sympathetic to.
>Lots of web pages have a bunch of blank lines near the top and scattered 
>elsewhere because they are generated by templates. "instructions" in the 
>template are omitted from the output, but the text content around the 
>instructions (including the newlines and indentation) do appear. Some 
>version of Java Servlets actually added a trimDirectiveWhitespaces 
>feature to help with this issue 
>(http://stackoverflow.com/questions/208736/strip-whitespace-from-jsp-outpu
>t), which reflects the problem.

But when do you see using a template to generate a JSON sequence?  I can't 
imagine when that would be the right choice from a software architecture 
perspective.

>> Also, could you describe why the consequences of an invalid value would 
>>be worse than I'm imagining?  Most of the reasons I can think of to use 
>>this format will likely ignore truncation errors.  

It's not a hack to ignore the error entirely, which is what I think will 
happen most often.

>Saying "don't send bare numbers" would be another special case

Yes, JSON is frankly awful with respect to numbers, all around.  If you 
cared at all about fidelity of representation vs. internal use, you might 
use something like CBOR, in which case you don't need to worry about RS or 
anything else to do sequences.

Since JSON numbers are a special case in the syntax of JSON, it doesn't 
surprise me that you need a special case in a stream of JSON numbers, if 
you want them to be dealt with correctly.

>Removing the text doesn't remove the risk of non-interoperability.

If you think it's a real risk, perhaps you could provide some better text?

-- 
Joe Hildebrand

---

From: Larry Masinter <[email-masked]>
Date: Mon, 22 Sep 2014 16:05:27 +0000

For me, it would have helped to point out that RS = control-^

I can type one into emacs using control-Q control-shift-6 (at least on a US keyboard).

Larry
--
http://larry.masinter.net