Subject: [secdir] secdir review of draft-ietf-json-text-sequence-11

From: Carl Wallace <[email-masked]>
Date: Sat, 13 Dec 2014 13:25:33 -0500

I have reviewed this document as part of the security directorate’s
ongoing effort to review all IETF documents being processed by the IESG.
These comments were written with the intent of improving security
requirements and considerations in IETF drafts.  Comments not addressed in
last call may be included in AD reviews during the IESG review.  Document
editors and WG chairs should treat these comments just like any other last
call comments.

This document describes a means of encoding and parsing arbitrarily large
sequences of JSON texts. The document claims no new security
considerations beyond those in RFC 7159.  I have some concern that the
addition of a LF by the JSON text sequence encoder that is not removed by
the JSON text sequence parser has the potential to break detached JSON web
signatures that cover JSON text sequence elements.  A few general comments
and questions are below:

- In the abstract of version 11 the hex value for ASCII line feeds was
changed to 0x1E from 0x0A.
- In section 2.1, shouldn’t possible-JSON be defined as *(not-RS) to allow
for parsing of <RS><RS><RS>?
- Section 2.3 should clarify that malformed JSON text sequences are also
not fatal (I.e., arriving at an RS without having seen a LF).
- The second paragraph of Section 2.3 suggests that an incremental JSON
parser may be used across content from multiple sequence elements.  The
ABNF for encoders does not seem to support this.
- The last sentence of the second paragraph in Section 2.4 is a bit
confusing. If a JSON-text has no trailing ws and the LF from the JSON text
sequence is used to match the ws rule, should this be reported as a
malformed JSON text sequence, a malformed JSON-text or neither?
- In section 2.4, the “parsers” referenced in the second and third
paragraphs should be qualified as JSON text sequence parsers or JSON
parsers.  
- In section 2.4, assuming the parser is a JSON text sequence parser, why
is the MUST drop in the third paragraph necessary given potential use of
incremental parsers mentioned in section 2.3 and later in 2.4? How do you
distinguish between a “non-self-delimited top-level value” and a piece of
JSON-text being parsed incrementally?
- The examples for possibly truncated JSON texts seem to assume that the
LF appended by a JSON text sequence encoder is absent. How would a
truncated JSON text within a properly delimited JSON text sequence be
detected? For example, if a component feeding JSON text elements to a JSON
text sequence encoder failed where the JSON text sequence encoder
succeeds, the result would be a truncated JSON text within a valid JSON
text sequence. I realize the ABNF should preclude encoding invalid JSON
texts but there is no text that instructs encoders to fail when invalid
JSON texts are detected.
- The failure to address the LF in the parser section seems like it could
cause problems for re-encoding.  What would prevent accumulation of LF
characters as a JSON text sequence was encoded, parsed, re-encoded,
re-parsed and re-encoded?

---

From: Nico Williams <[email-masked]>
Date: Mon, 15 Dec 2014 18:01:13 -0600

> ongoing effort to review all IETF documents being processed by the IESG.

Thanks.

> considerations beyond those in RFC 7159.  I have some concern that the

Though it turns out that RFC7159's security considerations section was
probably too brief.  Anyways.

> signatures that cover JSON text sequence elements.  A few general comments

That's a good point.  Signing individual JSON text sequence elements is
outside the scope of this document.  Either a complete sequence is
signed, or individual texts are signed with a wrapper.  I wouldn't
recommend transferring a sequence of texts and a parallel sequence of
signatures, say.

> changed to 0x1E from 0x0A.

Yeah, a widely noted typo :/

> for parsing of <RS><RS><RS>?

The preceding 1*RS in JSON-sequence takes care of that, although that
assumes greedy ABNF matching (RFC5234 does not speak to greediness).

> not fatal (I.e., arriving at an RS without having seen a LF).

That's covered in detail in section 2.4.  I wouldn't want to replicate
any part of 2.4 in 2.3.

> ABNF for encoders does not seem to support this.

Are you thinking of a a sequence like <RS> <JSON-text-w/o-trailing-LF>
<JSON-text> ... being parsed as two JSON texts?

Or are you thinking of a <RS> <JSON-text-containing-RS> <LF>?

The latter must yield a parse error from the JSON text parser.

As to the former... I think I'd thought about it and left it as-is,
though I can't recall why, with the security considerations text about
smuggling being sufficient (except it's not quite).

How about this new text:

   After reading and parsing a JSON text, the sequence parser MUST skip
   to the RS byte following the JSON text (or terminate if at end of
   input).

And in section 3:

   JSON text sequences like <RS> <JSON-text> <JSON-text> <LF> are
   sequences of one text, not two.  JSON text sequence parser
   implementations may emit the first JSON text, but should detect and
   log such sequences.

> malformed JSON text sequence, a malformed JSON-text or neither?

Neither.  The LF doubles as the ws that makes the number/boolean/null
parse correctly, and as the LF needed for the JSON text sequence.

This is also true for string/array/object, but since those are
self-delimited the presence of the LF is less critical.

Section 2.4 is really

a) justification for the LF,

b) a warning to implementors not to include a bug where permissive
   parsing (not insisting on the trailing LF) results in very incorrect
   results when the top-level value type is a number (or boolean, or
   null).

> paragraphs should be qualified as JSON text sequence parsers or JSON

When used without a qualifier the intended meaning is "JSON text
sequence parser", but that's a mouthful.  I may add text in section 1.1
about this.

> JSON-text being parsed incrementally?

See above.  In my implementation I check that the last byte consumed by
the [incremental] JSON text parser was a whitespace character.

> detected? [...]

Well, if the LF is present then it's not truncated.  E.g., the number in
<RS>1<RS> may have been truncated, but the number in <RS>123<LF> cannot
have been.

>     [...] For example, if a component feeding JSON text elements to a JSON

Yes, a JSON text sequence encoder should also encode the texts
themselves.  When it doesn't the result may be garbage, but the JSON
text sequence parser can cope with such garbage.

> re-parsed and re-encoded?

There's no requirement that parsing and re-encoding be the identity
function.  There's no such requirement for JSON texts, therefore there
can't be for JSON text sequences.

This comes up a lot for JSON implementations.  Users often want the
result of parsing and re-encoding to leave numbers as-is, object [key]
names in the same order as in the input, indentation/whitespace the
same, and so on.  The answer is generally "sorry, that's ETOOHARD".

(There are implementations that preserve object name/value insertion/
parse order.  But keeping track of encoded number form along with parsed
numbers can be a headache.  Besides, implementations often lose number
precision/range.)

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Tue, 16 Dec 2014 07:51:32 -0500

Inline...

On 12/15/14, 7:01 PM, "Nico Williams" <[email-masked]> wrote:

><snip>

Seems a good place to augment since you have noticed shortcomings.

>recommend transferring a sequence of texts and a parallel sequence of

It’s probably worth stating up front the process of encoding a JSON text
sequence alters the JSON text sequence elements and that signing
individual elements is outside the scope.

><snip>

OK.

>That's covered in detail in section 2.4.  I wouldn't want to replicate

I don’t recall seeing any mention of malformed JSON text sequences. I
think you either get to a next <RS> or end of the stream.

><JSON-text> ... being parsed as two JSON texts?

I was thinking more like <RS> <some fraction of JSON possibly a single
character>.

>   to the RS byte following the JSON text (or terminate if at end of

I think this text would be a nice addition. I noticed the lack of calling
out "read until end if no <RS>" but it seemed implied.

>parse correctly, and as the LF needed for the JSON text sequence.

Is this doubling simply trying to save a byte or something else?

In any case, the JSON text sequence parser cannot know whether the LF is
serving as the ws for the entire number that was to have been encoded or a
truncated value. It seems better for the JSON text sequence encoder to
encode what it gets as a sequence, the JSON text sequence parser to remove
what the encoder added with the JSON parser left to figure out what to do
with the “possible JSON” and/or error indicating a sequence parsing error.

Reporting possible truncation would be just as easy when finding a
<RS>123<LF><RS> instead of <RS>123<ws><LF><RS>. If the only purpose of the
<LF> is to attempt to serve in place of missing <ws>, simply parsing from
<RS> to <RS> seems fine. Given a missing <ws> is not the only form of
possible truncation, it doesn’t buy much to add the <LF>.

>   results when the top-level value type is a number (or boolean, or

As noted above, the warning about possibly truncated JSON text is
necessary, not sure about a).

><snip>

I think this is where we differ. How can the JSON text sequence decoder
know whether or not 123 was the complete value that should have been
encoded in the sequence? The component that generated 123 may have been
intended to pass 1234<ws> before it failed. In this case, the JSON text
sequence encoder is not doing a favor by supplying the <LF> as the <ws>.
I would prefer for the JSON text sequence encoder to not alter what it is
encoding by lending the <LF> to the element.

>text sequence parser can cope with such garbage.

This gets at my question about the ABNF not supporting incremental
encoding. An incremental snip may not parse (for any number of reasons) so
the sequence encoder has a bit of state to maintain. In any case, calling
out this “should also encode" for encoders would be a helpful addition.
This probably causes some interop problems worth mentioning where one
encoder just encodes what it is handed without re-encoding, possibly in
support of incremental encoding, while another (subsequently used) encoder
does try to parse each element.

>numbers can be a headache.  Besides, implementations often lose number

OK, though as noted above I still don’t see the need for adding the <LF>
in the encoder without removing it in the corresponding parser.

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Dec 2014 10:32:42 -0600

> Seems a good place to augment since you have noticed shortcomings.

Indeed.

> >> [...]

OK, that will be section 3 (security considerations text), something
like:

   Parsing and re-encoding a JSON text sequence need not produce the
   same sequence of octets.  Do not rely on being able to reproduce the
   same inputs to a cryptographic integrity protection function.

> think you either get to a next <RS> or end of the stream.

Section 2.3 actually says "malformed" in its first sentence.  It
mentions truncation only as an example of why a JSON text might be
malformed, in the second sentent.

There is also mention of malformed JSON texts in section 2.4 (explaining
why 'true' must not be produce upon merely consuming the 'e' in it, as
the next octet might not be a ws, nor RS).

Here's the proposed change:

OLD:
2.3.  Incomplete JSON texts are not fatal

NEW:
2.3.  Incomplete or malformed JSON texts are not fatal

(and s/textm/text,/)

> I was thinking more like <RS> <some fraction of JSON possibly a single

The ABNF for encoding assumes proper operation, yes.  The parser ABNF
takes care of malformed JSON texts.  This may be a simplification, but I
think it's a useful one.

OK.

> [extensive discussion of the LF elided]
>

There is no need to remove it in the sequence parser, though the
sequence parser may do it.  The sequence parser need only reject
top-level number/true/false/null values whose text did not end in a
whitespace.  A sequence parser could do this by insisting that the
sequence element end in an LF, and it can remove the trailing LF as well
as leaving it in, as the trailing LF's presence (or absence) does not
affect the validity of the JSON text to be parsed.

Adding the LF in the sequence encoder does not hurt, since all JSON
texts can end with arbitrary amounts of ws.  It merely helps delimit
sequence elements, both to ensure that top-level numbers/true/false/null
are delimited, and to help keep lines shorter for users using $PAGER and
$EDITOR to view JSON text sequences.

Delimiting otherwise non-self- delimiting texts is an important
function.

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Tue, 16 Dec 2014 12:20:08 -0500

>   same inputs to a cryptographic integrity protection function.

If supporting signing is not important to anyone, OK. This seems like a
significant sacrifice, especially when positioned against the benefit of
adding but not removing <LF>s.

>malformed, in the second sentent.

I am making a distinction between failure to parse a JSON text and failure
to parse a JSON text sequence. I think the text only addresses the former.

>> [extensive discussion of the LF elided]

How can a decoder know that <RS>123<LF><RS> was what the originator
intended and not something that was terminated by the text sequence
encoder? The originator may have intended <RS>1234<ws><LF><RS>. There
seems to be some assumption that the supplier of JSON text may fail to
self-delimit but would not fail to supply the full value. It’s a contrived
example, but how should an incremental JSON parser handle texts returned
from a parser operating on the sequence: <RS>123<LF><RS>4<ws><LF><RS>?
Would it be two values 123 and 4 or one value 1234? Why is it not be
preferable to report an error here <RS>123<LF><RS> instead of trying to
auto-terminate it when encoding the sequence?

>affect the validity of the JSON text to be parsed.

Is this universally true where JSON text is incrementally packaged into a
text sequence?

>Delimiting otherwise non-self- delimiting texts is an important

I guess we just disagree on whether the text sequence encoder is
necessarily in a position to terminate data that may be incrementally
supplied or incompletely supplied by a caller and whether or not this
important function should be allocated to the caller instead of to the
JSON text sequence encoder.  One alternative would be to add a <ws> only
when a non-self-delimited text is passed to a non-incremental encoder,
then encode that altered value into the sequence (and terminate all
sequence elements with <RS> only).

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Dec 2014 11:48:34 -0600

> adding but not removing <LF>s.

Supporting validation of signed sequences by first re-encoding the
sequence is absolutely not a goal.

This is almost dicta for all encodings of any messages.

We found out long ago that it doesn't work when the encoding does have a
canonical form (and even when it does but it's just not used by the
signer).

> to parse a JSON text sequence. I think the text only addresses the former.

The whole section is about JSON text parse errors not being fatal for
sequence parsing.  I don't understand the objection.  Perhaps if you
propose text I will?

> auto-terminate it when encoding the sequence?

The assumption is that the "process" writing the sequence will properly
encode the sequence elements, and will write the <RS><element><LF>
sequence correctly.  There is no assumption about atomic completion of
the write.  There is an assumption that incomplete writes will be
truncated from some arbitrary point in that byte sequence to the end of
it (that is, bytes will not be dropped from the middle or beginning).

The concerns about truncation spring from limitations of POSIX write
semantics, particularly O_APPEND writes.  Applications may have to use
writev() or else marshall the <RS><element><LF> into a buffer prior to
calling write().  These details are out of scope.

Applications will have to synchronize if they use write() to write the
RS, then an incremental JSON text encoder that may call write() multiple
times, then again a write() to write the LF.  This too is out of scope.

A paragraph of text about these assumptions may be warranted, but I
really don't want to have any references to POSIX and so on.  I think
the need for a modicum of atomicity (that which POSIX write semantics
provide) should be evident to implementors.

> Is this universally true where JSON text is incrementally packaged into a

See above.

> JSON text sequence encoder.  One alternative would be to add a <ws> only

Of course a properly functioning encoder on a properly function system
is in a position to terminate each element.  How can this be in doubt?

A sequence encoder might write() RS, then invoke an incremental JSON
text encoder to encode and write() the JSON text, then finally when the
JSON text encoder completes its task, the sequence encoder write()s the
LF.

The sequence encoder can also marshall the whole thing into a buffer and
write() that.

Part of the point of JSON text sequences is to permit online processing
of large data sets without requiring a streaming JSON text parser (which
is not the same as an incremental parser).  That point also applies to
encoding: if the sequence elements are small enough to fit into memory
and be parsed non-incrementally, then they also necessarily meet similar
constraints on the encoder side.

The only way to screw this up is when breaking down the process of
writing a sequence into a non-atomic sequence of operations that race in
a multi-process/threaded writer.  The details of that problem and how to
avoid it are clearly out of scope for this document.

> sequence elements with <RS> only).

The RS/LF bracketing was the result of lengthy discussions on the JSON
WG list.  The RS/LF bracketing design should not be reconsidered at this
time unless you have a security concern that cannot be addressed
otherwise.  I reject the concern about validation of signatures via the
use of re-encoded sequences (see above and earlier) -- it is commonly
accepted and strongly recommended practice that signatures should be
validated over what is signed, then and only then (after the signature
is validated successfully) should the payload be parsed.  If you have
any other security concerns relating to the LF, let's hear them.

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Tue, 16 Dec 2014 13:46:52 -0500

Inline...

>sequence is absolutely not a goal.

Re-encoding is not the point. The value parsed from the sequence has an
extra <LF> relative to what was passed in and would thus break a signature
generated on the original value. I don’t think there’s any lack of clarity
on this point. You already noted that in your view either the entire
sequence must be signed or individual elements signed with a wrapper.  I
should have written “if supporting detached signing is not important to
anyone, OK”.

>sequence parsing.  I don't understand the objection.  Perhaps if you

I think given the lending of <LF>s to the JSON text, there is not such
thing as a JSON text sequence parsing error. You find an RS to terminate
an element or you run out of bytes and terminate the element - no failures
at the sequence parsing level (though there may be errors that percolate
to the JSON parser).

>> auto-terminate it when encoding the sequence?

You have avoided this question in a couple of forms now. An answer here
would probably clarify things tremendously with regard to <LF> additions
and how incremental parsing or detecting incomplete encoding is supposed
to work.

>encode the sequence elements,

My assumption was “a" process encoding a “sequence" may receive an
"encoded sequence element” from “another” process, possibly as an API call
from a different box.

>and will write the <RS><element><LF>
>sequence correctly.

The point is what if element is incomplete, either due to failure or
incremental contribution to the sequence?

><POSIX discussion elided>
> This too is out of scope.

Which is fine because it is not the point.

>is in a position to terminate each element.  How can this be in doubt?

The JSON text encoder may be on one system and the JSON text sequence
encoder may be on another system (one of the examples in the draft is for
logs, so this must already be assumed as possible). In the example above,
if the text encoder fails after sending 123, the text sequence encoder
will add an <LF> and a decoder will not detect truncation. How can this be
in doubt?

>JSON text encoder completes its task, the sequence encoder write()s the

This may be the source of confusion. Does incremental parsing encompass a
single text sequence element or multiple text sequence elements?  Or can
it be either way?

>use of re-encoded sequences (see above and earlier)

I never associated re-encoded sequences with signature verification. I
only asked about <LF> accumulation during re-encoding.

>-- it is commonly

No additional security concerns relating to the LF other than what has
been stated (non-support for detached signatures, potential to alter
interpretation of elements in some circumstances). As noted above,
re-encoding is not and has not been the point re: signature verification.

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Dec 2014 13:37:09 -0600

> should have written “if supporting detached signing is not important to

The answer is clear: either wrap-to-sign, or remove the LF prior to
validation (since it must have been added by the encoder; if the text
was truncated, then the signature should fail).  Or, if you're chaining
encoding, parsing, encoding, parsing, you must canonicalize (except that
there is no JSON canonical form), or wrap-to-sign, or not sign, or not
re-encode.  These choices fall out from the fact that JSON has no
canonical encoding form.

If the encoder should have written <RS>123<LF><SP><LF><RS>4<LF> but
wrote <RS>123<LF><RS>4<LF>, then this would parse to the same two
elements (123 and 4), but any signatures of the first element taken over
123<LF><SP> would fail to validate.  That's too bad.

> thing as a JSON text sequence parsing error. You find an RS to terminate

Sure there is.  For example: <RS>{<LF><RS> (the LF might have been part
of the truncated JSON text).  Complying encoders shouldn't produce this,
but if they are writing log-style to a file, they might.  Which is why
the parse ABNF is more tolerant than the encode ABNF.

> an element or you run out of bytes and terminate the element - [...]

Pete Resnick, for example, wants the SHOULD NOT in section 2.3 to be
changed to something less strong.  My interpretation is that some people
want to parse JSON text sequences in contexts where failure to parse a
JSON text in the sequence should yield a failure to parse the sequence.

Therefore there is such a thing as a JSON text sequence parsing error.

>                                                        [...] - no failures

The application is on top, the JSON text parser at the bottom.  No
errors percolate to the JSON text parser :)

> >> from a parser operating on the sequence: <RS>123<LF><RS>4<ws><LF><RS>?

I think we're miscommunicating.

The correct way to parse <RS>123<LF><RS>4<ws><LF><RS> is as two octet
strings containing putative JSON texts: 123 and 4<ws> -- keeping
the LF or not is irrelevant (except to your signature validation
concern).  Each of those then parses to the numbers 123 and 4,
respectively.

Therefore the JSON parser should not see <RS>123<LF><RS>4<ws><LF><RS>,
it should see 123 (or 123<LF>) and then 4<ws> (or 4<ws><LF>).  Feeding
the LF, or not, to the JSON text parser is not important, since it
shouldn't require it.

One might feed the entire sequence to the incremental JSON text parser
and rely on it complaining about <RS>, skipping past it, resetting the
parser's state, and restarting it.  That's a perfectly legitimate
implementation design.

One might also scan for <RS>, split on <RS>, and feed the resulting
possible-JSON texts to the JSON text parser.

> >> Would it be two values 123 and 4 or one value 1234? Why is it not be

123 and 4.

> >> auto-terminate it when encoding the sequence?

I don't understand this question.

> and how incremental parsing or detecting incomplete encoding is supposed

LF is there for several reasons, one of them being to unambiguously
terminate non-self-delimiting top-level values.

> "encoded sequence element” from “another” process, possibly as an API call

I answered that earlier (right?) and then lost track of this concern.  A
process that merely adds RS/LF bracketing without parsing the sequence
elements... can produce invalid sequences, and might be encoding
sequence elements that are themselves sequences (though this is easy to
check for: just scan for RS in the input).

Although such an encoder is not considered in the I-D, we could add some
text about it.  I'd rather say: don't do that, always encode the JSON
texts to encode the JSON text sequence.

> incremental contribution to the sequence?

Section 2.1 says it's a possible JSON text.  Section 2.3 tells you what
to do if it's incomplete.  Section 2.4 explains how to determinie if
<element> is complete when it's not a self-delimiting value (and, of
course, if it is self-delimiting, then it is self-delimiting, but the LF
is still useful for the other reasons that I gave).

> Which is fine because it is not the point.

If I still haven't answered your question then please propose some text.

> will add an <LF> and a decoder will not detect truncation. How can this be

It's not.  I just don't think one should encode JSON text sequences this
way.  After lunch I'll propose text explicitly requiring sequence
encoders to also invoke the JSON text encoder.

See above.

> only asked about <LF> accumulation during re-encoding.

If you parse a sequence and its elements, and re-encode, you'll get more
differences than different amounts of LFs.

OK.

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Tue, 16 Dec 2014 15:44:05 -0500

On 12/16/14, 2:37 PM, "Nico Williams" <[email-masked]> wrote:

>><snip>>Of course a properly functioning encoder on a properly function
>>system

Most of the words in this now very long thread stem from this. I will wait
to review new text. I don’t think there is any lack of clarify on the
signature issue. Most of the remainder is related to the document’s
recognition of <RS>123<RS> as a detachable truncation problem but not
<RS>123<LF><RS> where the truncation occurred prior to invoking the
sequence encoder.

---

From: Nico Williams <[email-masked]>
Date: Tue, 16 Dec 2014 15:35:39 -0600

> <RS>123<LF><RS> where the truncation occurred prior to invoking the

In section 2.2, add before the last paragraph:

   JSON text sequence encoders are expected to ensure that the sequence
   elements are properly formed. When the JSON text sequence encoder
   does the JSON text encoding, the sequence elements will naturally be
   properly formed. When the JSON text sequence encoder accepts
   already-encoded JSON texts, the JSON text sequence encoder ought to
   to parse them before adding them to a sequence.

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Tue, 16 Dec 2014 21:13:55 -0500

On 12/16/14, 4:35 PM, "Nico Williams" <[email-masked]> wrote:

>> recognition of <RS>123<RS> as a detectable truncation problem but not

I don’t think this is quite there as parsing the already encoded texts
does not necessarily detect truncation. Even where the text sequence
encoder does the JSON text encoding it may do so on an incomplete value
due to crash or administrative process termination depending on the how
the JSON text sequence parser is situated relative to the JSON text
element source.  I am at a loss to supply alternate text because the text
would cement an arrangement of the encoders that is probably unnecessarily
rigid for a document defining a data structure. I still think the solution
is to remove the delimiters added by the JSON text sequence encoder in the
JSON text sequence decoder.  This seems cleaner to me.  It would probably
require the encoder to reject inputs that have not been properly
terminated or perhaps have a flag to auto-add <ws> to non-self-delimited
top level values before adding the <LF> where such is safe to do.

As you have noted, the existing text that has been agreed to by the WG so
I will defer to you, the WG and the ADs on the <LF> point. I think you
already agreed to add some text highlighting the fact that text sequence
elements are poor targets for detached signatures due to the fact that
encoding the element in the sequence changes it.

---

From: Carl Wallace <[email-masked]>
Date: Wed, 17 Dec 2014 06:51:41 -0500

Sorry for the self-reply, there was also some additional text to add
clarifying incremental parsing does not cut across multiple JSON text
sequence elements (in addition to a note regarding fitness for detached
signatures and any <LF> notes).

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Dec 2014 12:21:37 -0600

> signatures and any <LF> notes).

That's strongly implied by the parsing ABNF.

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Dec 2014 12:55:28 -0600

On Tue, Dec 16, 2014 at 09:13:55PM -0500, Carl Wallace wrote:
>                                          [...]. I still think the solution

Tolerating a missing LF seems like a fine thing to do if the top-level
value was nonetheless valid and delimited.

On the other hand it adds some ambiguity if some sequence parser
implementations can tolerate it and others can't.

For logging applications tolerating a missing LF seems like a desirable
thing to do, actually.  For non-logging applications not tolerating it
seems better.

Does requiring that the sequence parser strip the trailing LF
complicate the sequence parser ABNF?  I think the stripping of the
trailing LF simply can't be expressed in ABNF, only in prose.

I'll play with an implementation tonight and see what I think after I
re-write some code (jq, specifically).  It may well be that I like it
better your way.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 17 Dec 2014 17:41:18 -0600

> implementations can tolerate it and others can't.

I continue to think that the best thing to do is have a recommendation
as to how to handle ambiguities, and point them out.

Therefore I don't think we should say that the parser MUST strip the
trailing LF, but SHOULD (or MAY not strip it) would be fine, and should
address the concern about alterations to JSON texts that can then affect
cryptographic integrity protection.

So, how about these changes to section 2.1:

    The ABNF [RFC5234] for the JSON text sequence parser is as given in
    Figure 1.

-     JSON-sequence = *(1*RS possible-JSON)
+     JSON-sequence = *(1*RS possible-JSON LF)
+     LF = %x0A; "line feed" (LF), see RFC20
      RS = %x1E; "record separator" (RS), see RFC20
               ; Also known as: Unicode Character 'INFORMATION SEPARATOR
               ;                TWO' (U+001E)
      possible-JSON = 1*(not-RS); attempt to parse as UTF-8-encoded
                                ; JSON text (see RFC7159)
      not-RS = %x00-1d / %x1f-ff; any octets other than RS

                      Figure 1: JSON text sequence ABNF

    In prose: a series of octet strings, each containing any octet other
-   than a record separator (RS) (0x1E) [RFC0020], all octet strings
-   separated from each other by RS octets.  Each octet string in the
-   sequence is to be parsed as a JSON text in the UTF-8 encoding
-   [RFC3629].
+   than a record separator (RS) (0x1E) [RFC0020], and each such octet
+   string prefixed by an RS octet and suffixed by an LF octet.  Each
+   octet string in the sequence is to be parsed as a JSON text in the
+   UTF-8 encoding [RFC3629].

    If parsing of such an octet string as a UTF-8-encoded JSON text
    fails, the parser SHOULD nonetheless continue parsing the remainder
    of the sequence.  The parser can report such failures to applications
    (which might then choose to terminate parsing of a sequence).
    Multiple consecutive RS octets do not denote empty sequence elements
    between them, and can be ignored.

+   JSON text sequence parsers MAY include the LF suffix (from the
+   sequence encoding) when parsing a JSON text sequence element, but it
+   is best not to when the sequence parser's output is the element JSON
+   texts rather than an internal representation of the parsed element
+   JSON texts.
+
    There is no end of sequence indicator.

?

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Thu, 18 Dec 2014 07:12:34 -0500

>> implementations can tolerate it and others can't.

The suggestion is not to tolerate a missing <LF> but to not always add
them in the text sequence encoder in the first place. The <LF> addition
would essentially be an extension of the JSON text encoder (albeit
implemented in the text sequence encoder).  There would be no ambiguity.
The parser would always leave it in. Where integrity mechanisms are used,
the auto-<LF> would need to be turned off and the source responsible for
properly terminating inputs to the text sequence encoder.

>cryptographic integrity protection.

Thinking more about stripping out the <LF>, that won’t always work either
for the same reason you have the parser reject non-self-delimited texts
that do not end with a <LF>. The text sequence encoder could terminate
before the <LF> is written possibly leaving a <LF> added by the source
exposed (and removed by a decoder). The parser really can’t know if the
JSON encoder or JSON text sequence encoder added a trailing <LF>.

---

From: Nico Williams <[email-masked]>
Date: Thu, 18 Dec 2014 18:43:10 -0600

> them in the text sequence encoder in the first place. The <LF> addition

They can be tolerated in the parser.  Logging applications that encode
sequences correctly might nonetheless be subject to truncation of
sequence elements (including RS and trailing LF bytes) while
incrementally encoding a sequence, followed by additional elements being
added later (again, O_APPEND...).

Aside: if we'd never considered truncation we also wouldn't have needed
RS bytes, just trailing LF bytes, and optional ones at that (which is
exactly what https://stedolan.github.io/jq implements and has for a long
time now).  The problem with truncation is that there was no way to
figure out how to resync a parser, therefore no way to make it tolerate
truncation, not without RS, which in turn makes RS-less sequences not so
useful for logging applications.  We could abandon this, but several
people expressed the need for this format only when this was addressed,
and I do believe using RS (or similar) is better than not.

> implemented in the text sequence encoder).  There would be no ambiguity.

No extensions of the JSON text encoder can be had here, not even
notional ones.  We're not updating RFC7159, not even implicitly.

Off-the-shelf JSON text parsers and encoders must be sufficient to
implement JSON text sequence parsers and encoders.

> The parser would always leave it in. Where integrity mechanisms are used,

Whether the parser leaves it in or not is unimportant.  What's important
is that the parser uses it to detect truncation of JSON texts that can
be truncated and still be valid but _different_ from the original.

As noted, ambiguities result from this looseness about how the trailing
LFs are handled.  These are noted in the security considerations section
in the upcoming -12.  That should suffice.

> properly terminating inputs to the text sequence encoder.

Integrity mechanisms are out of scope in this document.

> JSON encoder or JSON text sequence encoder added a trailing <LF>.

When the sequence parser knows that the encoder of a sequence wasn't
subject to truncation, then the parser can indeed require and remove the
trailing LF.  In other cases it can't.

As JSON texts can have arbitrary amounts of trailing whitespace,
integrity mechanisms are out of scope, and JSON texts have no canonical
form (and are subject to being changed when parsed then re-encoded
without meaningful modification to the parsed value), there's no need to
concern ourselves further with this matter.

The upcoming -12 notes the ambiguities and lack of integrity mechanism
support.  I propose no further changes regarding how and whether to
detect if a trailing LF was added by a sequence encoder or not, or
whether to pass it to the JSON text parser.

Nico
--

---

From: Carl Wallace <[email-masked]>
Date: Fri, 19 Dec 2014 07:03:18 -0500

On 12/18/14, 7:43 PM, "Nico Williams" <[email-masked]> wrote:

><large snip>

OK

><large snip>

OK

---

From: Carl Wallace <[email-masked]>
Date: Wed, 24 Dec 2014 15:42:59 -0500

The new text in the security considerations creates a misimpression that
signature problems require “repeated parsing and re-encoding”.  I suggest
the following mods to the current text to remove this misimpression:

Current:
Repeated parsing and re-encoding of a JSON text sequence can result in the
addition (or stripping) of trailing LF bytes from (to) individual sequence
element JSON texts.  This can break signature validation.  JSON has no
canonical form for JSON texts, therefore neither does the JSON text
sequence format.

Suggested:
Encoding a JSON text sequence will result in the addition of a trailing LF
byte to individual sequence element JSON texts.  This can break signature
validation.  JSON has no canonical form for JSON texts, therefore neither
does the JSON text sequence format.