Subject: Re: [Json] NUDGE: Re: Second WG Last Call for json-text-sequence

From: Nico Williams <[email-masked]>
Date: Mon, 8 Sep 2014 22:43:19 -0500

On Thu, Aug 28, 2014 at 05:13:06PM +0000, Joe Hildebrand (jhildebr) wrote:
> On 8/28/14, 10:05 AM, ""Martin J. Dürst"" <[email-masked]> wrote:
> 
> >Also, there is still a lack of confirming comments (something along the 
> >lines of 'I read it - please ship it'). Even the above repeated begging 
> >from Paul to help the chairs determine consensus has only let to more 
> >bikeshedding.
> 
> In this case, one person's bikeshed is another person's technical concern. 
> Once we get the ABNF straightened out, I'm a strong +1 on publishing this 
> document.

Alright, for the ABNF I propose two definitions, one for the parser:

    sequence = *(1*RS thing 1*RS)   ; <thing> contains a JSON-text
    RS = %x1E
    thing = *all-but-RS-chars
    all-but-RS-chars = %x00-%x1D / %x1F-%xFF

and one for the encoder:

    sequence = *(RS JSON-text RS)
    JSON-text = <given by RFC7159>

This would yield a reusable sequence scheme that could work for XML too,
if anyone wanted it.  It might be useful to change the MIME type from
application/json-seq to application/seq-json so that a putative future
application/seq-xml fits right in.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Sep 2014 07:51:31 +0200

On 2014-09-09 05:43, Nico Williams wrote:
> ...

Almost.

RFC 7159 uses ABNF to describe sequences of characters, not octets. If 
you want to mix this ABNF with RFC 7159 productions, you'd need to be 
consistent with that. For instance:

    all-but-RS-chars = %x00-1D / %x1F-10FFFF

(and you've got the ABNF range syntax wrong)

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 09:43:36 -0500

On Tue, Sep 09, 2014 at 07:51:31AM +0200, Julian Reschke wrote:
> RFC 7159 uses ABNF to describe sequences of characters, not octets.
> If you want to mix this ABNF with RFC 7159 productions, you'd need
> to be consistent with that. For instance:
> 

The idea is that between the RS brackets there's just octets.  You find
those and then interpret them as JSN-texts.  They might not be
well-formed, and they might not even be valid UTF-8 due to the logfile
unreliable append issues.

> (and you've got the ABNF range syntax wrong)

Yes:

    all-but-RS-chars = %x00-1D / %x1F-FF

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Sep 2014 16:52:55 +0200

> unreliable append issues.

I get that and I agree with it.

The trouble is that this spec uses ABNF on the octet level (ok), the 
JSON spec uses it on the character level (also ok), and you propose to 
mix productions from both specs -> trouble.

So minimally, this:

would need to become something like:

       sequence = *(RS JSON-text RS)
       JSON-text = <any octet sequence that encodes a RFC 7159 JSON-text 
using one of the allowed character encodings>

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 10:03:28 -0500

On Tue, Sep 09, 2014 at 04:52:55PM +0200, Julian Reschke wrote:
> JSON spec uses it on the character level (also ok), and you propose
> to mix productions from both specs -> trouble.

No, I'm not mixing them, not on the parser side.  You parse the sequence
to get the texts, then you *separately* parse the texts.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Sep 2014 17:17:03 +0200

>      JSON-text = <given by RFC7159>

*does* mix them (for the encoder). I read it as if I could use the 
JSON-text production from <http://rfc7159.net/rfc7159#rfc.section.2>, 
and that's not the case, right?

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 10:22:00 -0500

> >     all-but-RS-chars = %x00-%x1D / %x1F-%xFF

There's nothing above about JSON-text, except in a comment :)

> JSON-text production from
> <http://rfc7159.net/rfc7159#rfc.section.2>, and that's not the case,
> right?

On the encoder side there's no all-but-RS-chars.  These are two distinct
rules.

The parser ABNF has no direct reference to any rules from RFC7159 -- it
deals only with octet strings.

The encoder ABNF does have a reference to RFC7159 and does not otherwise
deal with any kind of string.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 9 Sep 2014 20:05:27 +0000

>application/seq-xml fits right in.

Leaving aside Julian's issues, which I don't yet have an opinion on, I 
thought we had talked about the trailing RS as being optional for many 
applications, only really required if you're sending bare numbers?  With 
that slight tweak, the parser would be:

    sequence = *(1*RS thing *RS)

and the encoder:

    sequence = *(RS delimited)
    delimited = (delimited-number / non-number-text)
    delimited-number = ws number ws RS
    non-number-text = ws non-number-value ws
    non-number-value = false / null / true / object / array / string
    ws,number,false,null,true,object,array,string = <given by RFC 7159>

(ew).  Or we could just explain that in the text.  Or we could say that 
encoders MUST include at least one trailing whitespace character if they 
write a bare number.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 15:19:35 -0500

On Tue, Sep 09, 2014 at 08:05:27PM +0000, Joe Hildebrand (jhildebr) wrote:
> applications, only really required if you're sending bare numbers?
> With that slight tweak, the parser would be:
> 

Yes, but it's much easier to just say "always bracket with leading and
trailing RS".  Then we don't have to say anything about EOF and numbers.

On a related note, the parser's ABNF does need to be this:

    sequence = *(1*RS thing) RS

or

    sequence = *1(1*RS thing) RS)

to account for the possibility of missing RSes (again, logfile
incomplete writes).  I.e., an encoder might write

    RS thing1 RS

then

    RS thing2 RS

but what actually gets written appears might well be:

    RS thing1 RS thing2 RS

which should still parse as a sequence of those two things.  A strict
parser using 'sequence = *(1*RS thing *RS)' would fail to find thing2.

> and the encoder:
> 
>     [...]
> 
> (ew).  Or we could just explain that in the text.  Or we could say
> that encoders MUST include at least one trailing whitespace character
> if they write a bare number.

If the WG wants this I'm fine with it.  Heck, I'll go with whatever the
WG wants.  But it strikes me as unnecessarily complex just to optimize
away one byte in the common case.  No?

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 9 Sep 2014 20:55:19 +0000

>parser using 'sequence = *(1*RS thing *RS)' would fail to find thing2.

are you saying the greediness of the *RS at the end would eat up all of 
the RS's, leaving none for the initial 1*RS?

*(1*RS thing) RS

is a little counter-intuitive, since you wouldn't implement it that way on 
the output side, but I'm fine with it.  Your other suggestion:

*1(1*RS thing) RS)

doesn't parse for me.  could you add an open paren to clarify, please?

>away one byte in the common case.  No?

As long as the file is parseable with only one RS between records, some 
people will likely generate the files that way, no matter what the 
generator section says.  I agree that my suggestion was the outer limit of 
complexity; hopefully we can come up with something better.

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 16:25:43 -0500

> the RS's, leaving none for the initial 1*RS?

For example, yes.

> the output side, but I'm fine with it.  Your other suggestion:

I'm already splitting out the encoder side...

> doesn't parse for me.  could you add an open paren to clarify, please?

Whoops:

  *1((1*RS thing) RS)

"A sequence consists of any number of members separated by RS; an empty
string is an empty sequence."

Otherwise an empty string/stream would not parse as an empty sequence :)

> generator section says.  [...]

Yes, and that's fine.

> complexity; hopefully we can come up with something better.

*nod*

I think we're almost there.

Thanks,

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 9 Sep 2014 22:45:20 +0000

>  *1((1*RS thing) RS)

I assume you meant:

1*((1*RS thing) RS)

>string is an empty sequence."

and here, "string" is ambiguous.  I think empty "thing" is better, once we 
decide what a thing is.

>Otherwise an empty string/stream would not parse as an empty sequence :)

I agree that:

RS RS RS true RS RS RS
and
RS true RS true RS

should both be valid streams with no errors generated.  What about:

RS <space> RS

?  I think that should probably generate whatever error reporting you do 
for malformed/truncated things.

>I think we're almost there.

After talking with Matt Miller a little in the office this afternoon, I'm 
a little concerned that we have specified a format that might be able to 
detect truncation on the parse side, but we don't specify how that might 
be done.  For non-numbers, it's pretty easy.  For bare numbers, you would 
currently have to count the RS's between records.  That might eventually 
be made to work (with a bunch of effort on our part), but I'm skeptical.  
I do think a non-RS whitespace at the end of bare numbers makes it much 
easier to detect their truncation.

Alternately, I'd be just fine warning folks never to expect to be able to 
determine when a number has been truncated, and using a different encoding 
such as [1], ["1"], or {"num": 1} if they care about that property.  At 
which point we can go back to a simpler grammar...

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 18:18:47 -0500

On Tue, Sep 09, 2014 at 10:45:20PM +0000, Joe Hildebrand (jhildebr) wrote:
>                                                 [...].  What about:

The parser could report detected truncation, but it should not stop.  It
should continue parsing, and that input would be an empty sequence.

> easier to detect their truncation.

Ah, yes, I see the point of the SP (I had a couple of weeks ago then I
forgot).  Suppose we write:

    RS 12345 SP RS

then

    RS 56789 SP RS

And suppose both get truncated so that what appears is:

    RS 123 RS 567 <EOF>

With the extra SP we get to tell that neither of those numbers is
complete.

OK, so, ignoring ABNF greediness, on the parser side we'd have:

    sequence = *(1*RS *all-but-RS ws)
    all-but-RS = %x00-1D / %x1F-FF
    ws = <given by RFC7159>

And on the encoder side we'd have:

    sequence = *(RS JSON-text ws)
    JSON-text = <given by RFC7159>
    ws = <given by RFC7159>

But can we ignore ABNF greediness?

I'm also willing to just have a warning about top-level numbers.  The
problem only really applies to resources written as logfiles, after
all.

Nico
--

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 9 Sep 2014 23:58:42 +0000

>should continue parsing, and that input would be an empty sequence.

Yes, absolutely.  A recoverable error.

>With the extra SP we get to tell that neither of those numbers is

Exactly.

>    all-but-RS = %x00-1D / %x1F-FF

At this point, you could probably just mandate a LF for the ws, and 
everyone would be happy...

(/me ducks the rotten fruit and vegetables being thrown by the chairs)

>But can we ignore ABNF greediness?

I don't see a greediness problem any more on the decoder side.  You 
greedily eat RS until you get not-RS, then go until you get RS or EOF.  If 
the previous character wasn't ws (or LF), generate a recoverable 
truncation error.  Else parse what came between the RS and the ws (or LF).

>problem only really applies to resources written as logfiles, after

Yes.  And I don't see myself ever writing plain numbers for those 
applications.  Does anyone else have an opinion?

-- 
Joe Hildebrand

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 19:09:59 -0500

> everyone would be happy...

That's what I'd like, sure.  But I'll take whatever it takes to finish.

> (/me ducks the rotten fruit and vegetables being thrown by the chairs)

:)

> truncation error.  Else parse what came between the RS and the ws (or LF).

Fair enough!

Nico
--

---

From: Stefan Drees <[email-masked]>
Date: Wed, 10 Sep 2014 07:09:15 +0200

> applications.  Does anyone else have an opinion?

besides the elegance of correctly logging pi unadorned in a single 
eternal line :-?. No from my side a single lined
+1
for avoiding bare top-level numbers as excuses for neither grammar nor 
procedural complications.

"Stefan Drees"