Subject: [Json] Richard Barnes' Discuss on draft-ietf-json-text-sequence-11: (with DISCUSS and COMMENT)

From: "Richard Barnes" <[email-masked]>
Date: Wed, 17 Dec 2014 06:35:17 -0800

Richard Barnes has entered the following ballot position for
draft-ietf-json-text-sequence-11: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to http://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
http://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

Section 2.4.: The logic for having a terminator character here seems
pretty fishy to me.  Can you explain the scenario in which this setup
actually results in truncated texts being detected?  What is the piece of
software that recognizes the truncation and chooses to omit the
terminator?  It seems like in the obvious design (source of JSON texts
feeding into sequence encoder), you just end up with (RS <truncated-JSON>
LF) anyway.  In order for this not to be the case, you need to have
something in front of the sequence encoder that recognizes truncation and
signals it to the encoder

Section 2.4.: If you're going to have a terminator character, why 0x0A in
particular?  That character is valid in JSON texts, so using that as a
terminator requires special processing on the JSON texts.  As with the
above, this seems to require an architecture in which the source of JSON
texts cannot be independent of the sequence encoder (which kind of
defeats the purpose of this supposedly lightweight sequence encoding).

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Section 2.2.: "textm"

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Dec 2014 09:58:49 -0600

Loggers that write(), possibly using O_APPEND.  There's no guarantee
under POSIX that any or all bytes written will reach stable storage, but
whatever bytes do reach stable storage will be from the beginning of the
buffer towards the end.

(There was a commit in the Linux kernel recently, perhaps a year ago,
that enabled posting SIGKILL to a process writing to a file with
O_APPEND.  The signal will interrupt the writer at a block boundary
offset in the file being written.  Other circumstances where one can
expect similar behavior on other operating systems includes ENOSPC.)

> terminator?  It seems like in the obvious design (source of JSON texts

None.

> LF) anyway.  In order for this not to be the case, you need to have

Not if the writer does a single write() or writev() to write the RS, the
text, and the LF.  And not even if it does multiple writes with a lock
and dies in the middle of writing any one of those things.

> signals it to the encoder

The idea is that the sequence encoder is also invoking the JSON text
encoder, rather than operating on already-encoded JSON texts.

> terminator requires special processing on the JSON texts.  As with the

It's not that special, since you can scan for RS bytes and check that
byte preceding the next RS is LF (or other ws; truncation isn't
necessarily a problem, just truncation that leaves you with an ambiguous
text).

> defeats the purpose of this supposedly lightweight sequence encoding).

I don't think this is true.  One could simply bracket already-encoded
texts with RS/LF -- the sequence encoder might not be certain that the
texts are properly formed, but assuming that they are then the
bracketing is sufficient.  But yes, the design in mind is that the
sequence encoder also invokes the JSON text encoder.

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Wed, 17 Dec 2014 17:30:07 -0500

Inline.

> encoder, rather than operating on already-encoded JSON texts.

So it seems like in order to detect truncation, it needs to be true that
the JSON text generator (the logger in the above example) is writes a 0x0A
character after the text.  This needs to be integrated into the thing
submitting JSON texts to the sequence, or at least something earlier than
any code that could cause truncation.  This is not a requirement for a
general JSON generator, so you need to state this requirement explicitly.

Proposed text for the top of Section 2.4.:
OLD:
"""
Parsers MUST drop JSON-text sequence elements consisting of
non-self-delimited top-level values that may have been truncated (that are
not delimited by whitespace).
"""

NEW:
"""
JSON text sequences use 0x0A as a "canary" octet to detect truncation.
Parsers MUST drop JSON-text sequence elements consisting of
non-self-delimited top-level values that do not end in a 0x0A character
(since they have been truncated).  Since JSON is not required to be
terminated with a 0x0A character, this implies that the JSON texts that are
input to a JSON text sequence MUST be suffixed with 0x0A before any point
where truncation may occur.
"""

> sequence encoder also invokes the JSON text encoder.

Looking at the parser ABNF, this makes sense (since the 0x0A is not
there).  Consider this point cleared.

--Richard

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Dec 2014 17:19:17 -0600

> the JSON text generator (the logger in the above example) is writes a 0x0A

Well, the sequence generator.  There's no need for the JSON text encoder
to do this, and we wouldn't want to impose such a requirement on the
JSON text encoder.

> any code that could cause truncation.  This is not a requirement for a

Truncation detection is for the JSON text sequence parser.

A sequence encoder that uses already-encoded JSON texts from other
sources cannot detect truncation in them (unless those texts could only
be taken from another JSON text sequence).

Such a sequence encoder has to be responsible for arranging for
truncation detection with the source of those already-encoded JSON
texts.  I could add text saying _that_, but I don't see how we can
provide that arrangement here.  We certainly couldn't impose a
requirement for a trailing LF on JSON text encoders, as that would mean
updating RFC7159.

> general JSON generator, so you need to state this requirement explicitly.

Isn't this explicit enough in section 2.2?

> JSON text sequences use 0x0A as a "canary" octet to detect truncation.

This first new sentence is fine.

> where truncation may occur.

The second new sentence (the last one here) is superfluous if you accept
the above responses as to sequence encoders using already-encoded JSON
texts.

> [...]

Thanks,

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Mon, 22 Dec 2014 11:48:37 -0500

> the above responses as to sequence encoders using already-encoded JSON

I think you're still missing my point here.  If I understand your use case
right, you're envisioning the following process:

JSON.stringify -> sprintf("\x1E%s\x0A", text) --(trunc)--> Append to
sequence

That's a particular processing chain, and I agree that the truncation
detection provides value in that case.  What I'm trying to say is that
there are other ways of encoding a sequence that meet the same production
ABNF, but which do *not* provide truncation detection.  Most obviously:

JSON.stringify --(trunc)--> sprintf("\x1E%s\x0A", text) --> Append to
sequence

That's why we need to say that the 0x0A needs to be added before truncation
can occur.  It's a critical requirement for truncation detection to work.

It's not redundant with 2.2, since it's an additional requirement on the
process of encoding (beyond the ABNF).

--Richard

---

From: Nico Williams <[email-masked]>
Date: Mon, 22 Dec 2014 12:00:33 -0600

> JSON.stringify -> sprintf("\x1E%s\x0A", text) --(trunc)--> Append to

Yes.

> can occur.  It's a critical requirement for truncation detection to work.

The source of potential truncation of interest in this document is
"append to a file".  Other sources of corruption (e.g., transmission
over an insecure and fragile transport protocol) don't merely stop at
truncation, and we can't do anything about those other forms of
corruption as "JSON text sequence" is not a cryptographic format.

How would append-to-file truncation occur in this second case?  Clearly
the file can't be a JSON text sequence, since otherwise a JSON text
sequence encoder would have been in a position to ensure the writing of
the LF.  The source of truncation has to have been an append to a file
meant to contain a *single* JSON text, but then the writer isn't a
logger, and can take additional steps to ensure atomic operation.
Therefore I don't think this case can really occur.

This feels like a turtles-all-the-way-down situation, resolved by noting
that either each sequence encoder in the chain invokves the JSON text
sequence encoder itself, or is otherwise parsing and re-encoding a
previously-encoded JSON text sequence.  We can't make the JSON text
encoder the last turtle, since that would mean updating RFC7159 here,
which we can't really consider.

All of this leads me to think that this is a use case to be considered
out of scope or recommended against.  The latter looks like a
recommendation (or requirement) that the JSON text sequence encoder also
be the one invoking the JSON text encoder.

I'd even rather discourage the use of top-level numbers, true, false,
and null, rather than agree to the text you proposed that I objected to.
Instead, if you still think we need to say anything more, I'd rather say
either that the second use case you mention above is out of scope.

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Mon, 22 Dec 2014 14:01:31 -0500

On Mon, Dec 22, 2014 at 1:00 PM, Nico Williams <[email-masked]>
wrote:
> "append to a file".

That assumption is not stated anywhere in the document.

> Other sources of corruption (e.g., transmission
> How would append-to-file truncation occur in this second case?

Wouldn't that arise if you had something file-like between the JSON
producer and the sequence encoder?  For example:
> json-logger >/tmp/log-pipe
> sequence-encoder </tmp/log-pipe

--Richard

>   Clearly

---

From: Nico Williams <[email-masked]>
Date: Mon, 22 Dec 2014 13:21:16 -0600

> That assumption is not stated anywhere in the document.

It's mentioned in section 2.3:

   Per- Section 2.1, JSON text sequence parsers should not abort when an
   octet string contains a malformed JSON text, instead the JSON text
   sequence parser should skip to the next RS.  Such a situation may
   arise in contexts where, for example, append-writes to log files are
   truncated by the filesystem (e.g., due to a crash, or administrative
   process termination).

(as it appears in -12, which here is the same as -11 + s/textm/text,/).

This (and only this) source of truncation was one reason we ended up
with RS in the format at all.  Otherwise all we'd need is a ws after
non-self- delimited top-level JSON texts.

The other reason is that the RS is also useful for implementations that
don't have an incremental JSON text parser (thus the sequence parser has
to find the entire possible-JSON-text octet string to feed to the JSON
text parser, and without RS or similar, it couldn't).

(There are applications using a JSON text sequence format with no RS and
even no LF or other ws, where all the texts are objects.  These
sequences look like this: {"a":"foo"}{"a":"bar"}...  A sequence parser
with access to an incremental JSON text parser can manage to parse this,
but if all you have is a parser that wants complete texts, then you
lose.  Whereas <RS>{"a":"foo"}<LF><RS>{"a":"bar"}<LF> can be parsed with
any kind of off-the-shelf JSON text parser.)

> > sequence-encoder </tmp/log-pipe

It depends on the OS.  For POSIX systems there's a write() size below
which writes to pipes are atomic, and above which they are not, and even
in the latter case, if there's only one writer there's no problem
unless the writer dies (which the reader probably can't detect, unless
it's the writer's parent process).

I think informative text warning that truncation that occurs between
JSON text encoding and JSON text sequence encoding cannot be detected.

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Mon, 22 Dec 2014 15:14:48 -0500

>    truncated by the filesystem (e.g., due to a crash, or administrative

"May arise" does not mean "this mechanism is only useful when".  See
suggested text below.

> it's the writer's parent process).

So you're saying it could arise :)

> JSON text encoding and JSON text sequence encoding cannot be detected.

Proposed for Section 2.3, after the above-quoted paragraph:
"""
The JSON text sequence structure described by the encoding ABNF allows
for the detection under certain circumstances.  Namely, if you view the text
sequence as a collection of "records" of the form (RS JSON-text LF), then
it is possible for a parser to recognize when a record has been truncated
(as described in Section 2.4).  This mechanism does not cover other cases,
e.g., if the JSON text is truncated before the LF octet is appended.
Implementations that rely on the JSON text sequence structure to detect
truncation MUST ensure that JSON texts end in LF before any point where
truncation might occur.
"""

--Richard

---

From: Nico Williams <[email-masked]>
Date: Mon, 22 Dec 2014 14:19:06 -0600

> So you're saying it could arise :)

Yes, I suppose so :)

> truncation MUST ensure that JSON texts end in LF before any point where

Alright, I'll take this.

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Mon, 22 Dec 2014 15:30:52 -0500

> > truncation MUST ensure that JSON texts end in LF before any point where

Great.  Let me know when that's in, and I'll clear.

--Richard

---

From: Pete Resnick <[email-masked]>
Date: Mon, 22 Dec 2014 14:53:08 -0600

On 12/22/14 2:14 PM, Richard Barnes wrote:
> for the detection under certain circumstances.

I assume there's a missing "of truncation" after "detection", eh?

> Namely, if you view the text

Very MUSTy of you. Are you equally OK with the following?

    Namely, if you view the text sequence as a collection of "records" of
    the form (RS JSON-text LF), then it is possible for a parser to
    recognize when a record has been truncated (as described in Section
    2.4).  However, this mechanism does not cover other cases.
    Importantly, truncation cannot be detected if the JSON text is
    truncated before the LF octet is appended.

Since we're shooting for something "informative".

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Richard Barnes <[email-masked]>
Date: Mon, 22 Dec 2014 16:05:54 -0500

> Since we're shooting for something "informative".

Speak for yourself :)

This actually seems very MUSTy to me, as in "You don't get what you want if
you don't do this."  And it seems like something such that it should be
possible to check whether an implementation complies.

--Richard

>
>
> pr
>
> --
> Pete Resnick <http://www.qualcomm.com/~presnick/> <http://www.qualcomm.com/~presnick/>

---

From: "Resnick, Pete" <[email-masked]>
Date: Mon, 22 Dec 2014 21:24:35 +0000

Well, you know I don't buy into this "using MUSTs for compliance" nonsense. And there's certainly no interoperability issue here. I will in the end leave it to the WG, but is your desire for a MUST a stylistic preference, or is it "I really think implementers are going to screw the pooch if it's not in there?

BTW: Did I get the earlier edit correct?

pr

Richard Barnes <[email-masked]> wrote:

On Mon, Dec 22, 2014 at 3:53 PM, Pete Resnick <[email-masked]<mailto:[email-masked]>> wrote:
On 12/22/14 2:14 PM, Richard Barnes wrote:
I think informative text warning that truncation that occurs between
JSON text encoding and JSON text sequence encoding cannot be detected.

Speak for yourself :)

This actually seems very MUSTy to me, as in "You don't get what you want if you don't do this."  And it seems like something such that it should be possible to check whether an implementation complies.

--Richard

pr

--
Pete Resnick <http://www.qualcomm.com/~presnick/><http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478<tel:%2B1%20%28858%29651-4478>

---

From: Richard Barnes <[email-masked]>
Date: Mon, 22 Dec 2014 16:31:34 -0500

>  Well, you know I don't buy into this "using MUSTs for compliance" nonsense. And there's certainly no interoperability issue here. I will in the end leave it to the WG, but is your desire for a MUS…
I think there's a non-negligible risk that an implementor will set up a
scheme like the one I proposed up-thread, and think they are getting
truncation detection when in fact they are not.

> Yep.

--Richard

>  This actually seems very MUSTy to me, as in "You don't get what you want
> if you don't do this."  And it seems like something such that it should be

---

From: Nico Williams <[email-masked]>
Date: Mon, 22 Dec 2014 15:59:56 -0600

On Mon, Dec 22, 2014 at 04:31:34PM -0500, Richard Barnes wrote:
> >  Well, you know I don't buy into this "using MUSTs for compliance"
> >  nonsense. And there's certainly no interoperability issue here. I
> >  will in the end leave it to the WG, but is your desire for a MUST a
> >  stylistic preference, or is it "I really think implementers are
> >  going to screw the pooch if it's not in there?
> 
> I think there's a non-negligible risk that an implementor will set up
> a scheme like the one I proposed up-thread, and think they are getting

I only yielded as to the MUST because I want this over with.  But I
don't like it because I'm not sure how to adhere to that MUST in an
implementation where the sequence and text encoders are as separated as
you (and the secdir reviewer) think they might be.

How about this text, in section 2.4, instead:

   Detection of truncation of non-self-delimited sequence elements
   (numbers, true, false, and null) is only possible when the sequence
   encoder produces or receives complete JSON texts.  Implementations
   where the sequence encoder is not also in charge of encoding the
   individual JSON texts should ensure that those JSON texts are
   complete.

This does make clear when a problem might arise, and what to do about
it.  The "should" could be a SHOULD, even a MUST if need be, though
frankly, I think it should suffice to have a warning (as is already
present in section 3) about lack of integrity protection.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 23 Dec 2014 14:51:56 -0800

Nico has published a -12 of the draft. As shepherd, I think the new wording added is probably in line with what the WG would have expected (clarifications but no new requirements). Richard: let us know if this wording addresses your concerns.

--Paul Hoffman