Subject: [Json] Gen-ART and OPS-Dir review of draft-ietf-json-text-sequence-09

From: "Black, David" <[email-masked]>
Date: Fri, 5 Dec 2014 14:51:04 +0000

This is a combined Gen-ART and OPS-Dir review.  Boilerplate for both follows ...

I am the assigned Gen-ART reviewer for this draft. For background on
Gen-ART, please see the FAQ at:

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Please resolve these comments along with any other Last Call comments
you may receive.

I have reviewed this document as part of the Operational directorate's ongoing
effort to review all IETF documents being processed by the IESG.  These comments
were written primarily for the benefit of the operational area directors.
Document editors and WG chairs should treat these comments just like any other
last call comments.

Document: draft-ietf-json-text-sequence-09
Reviewer: David Black
Review Date: Dec 5, 2014
IETF LC End Date: Dec 5, 2014
IESG Telechat date: Dec 18, 2014

Summary: This draft is on the right track, but has open issues
 		described in the review.

This draft specifies a format that packs multiple JSON texts into a
single string.  The ASCII RS (0x1E) character is used to separate texts,
and a linefeed is appended to each text to ensure that a complete text
always ends with a whitespace character.

All of the open issues are minor - the most important ones center on
treatment of incomplete JSON texts - that appears to be an afterthought
in this draft and needs more attention.  I also found a couple of
minor issues in the Security and IANA Considerations sections, both of
which are almost nits.

Major issues: None.

Minor issues:

[A] Section 2.1:

   If parsing of such an octet string as a JSON text fails, the parser
   SHOULD nonetheless continue parsing the remainder of the sequence.

That's not quite right - there are two levels of parsing, JSON
sequence parsing and JSON text parsing of each text in the sequence,
both of which might be implemented in a single-pass parser.  For such an
implementation, the above sentence could be (mis-)read to imply that the
JSON text parse should resume from the point at which it failed, which
would be silly (although I've seen heroic PL/1 parsers do exactly that).
Instead, the parse needs to skip ahead to the next RS, ignoring the rest
of the JSON text that failed to parse.  I suggest:

   If parsing of such an octet string as a JSON text fails, and the
   octet string is followed by an RS octet, the parser
   SHOULD nonetheless skip ahead to that RS octet and continue parsing
   the remainder of the sequence from there.

That also covers the case where there is nothing more to parse after the
JSON text that caused the parse failure.

[B] Section 2.3:

Is incremental parsing of a JSON text within a sequence allowed, or
is the parser required to not produce any results until the parse of
the entire text is successful?  I'd expect that incremental parsing
is ok (so results may be produced from a text that ultimately fails
to parse), and I think that's worth stating.

[C] Section 2.4:

   Parsers MUST check that any JSON texts that are a top-level number
   include JSON whitespace ("ws" ABNF rule from [RFC7159]) after the
   number, otherwise the JSON-text may have been truncated.

That reference to the "ws" rule doesn't get the job done because that
rule allows a match to no characters - it's of the form ws = *( ... )
where ... is the list of whitespace characters.  What's needed here is
a rule of the form vws = 1*( ...) to force there to be at least one
whitespace character, but see the next issue for a better way to deal
with this topic by pulling the appended LF into the sequence parse
instead of the text parse.

[D] I wonder whether the possibility of incomplete texts ought to be
encoded into the parsing rules to directly catch JSON texts that must
be incomplete because the last character is not LF, e.g.:

     JSON-sequence = *(1*RS (possible-JSON / truncated-JSON / empty-JSON))
     RS = %x1E; "record separator" (RS), see RFC20
     possible-JSON = 1*(not-RS) LF ; attempt to parse as UTF-8-encoded
                               ; JSON text (see RFC7159)
     truncated-JSON = *(not-RS) not-LFRS); truncated, don't attempt
					; to parse as JSON text
     empty-JSON = LF ; only the LF appended by the encoder, nothing to parse

     not-RS = %x00-1D / %x1F-FF; any octet other than RS
     not-LFRS = %x00-09/ %x1B-1D / %x1F-FF; any octet other than RS or LF

Note that this won't detect all incomplete JSON texts, because LF is allowed
within a JSON text (and this should be stated).

[E] Section 3 - Security Considerations

Incomplete and malformed JSON texts can be used to attack JSON parsers -
that should be pointed out, as I don't see that in RFC 7159's security
considerations and incomplete texts are a relevant consideration for
this draft.

[F] Section 4 - IANA Considerations

   Security considerations: See <this document, once published>,
   Section 3.

   Interoperability considerations: Described herein.

   Published specification: <this document, once published>.

   Applications that use this media type: <by publication time
   <https://stedolan.github.io/jq> is likely to support this format>.

Replace all three instances of the angle bracketed text.  The first two
instances should be RFC references (e.g., RFC XXXX) w/a note to the RFC
Editor to insert the number of the RFC when published.  The third instance
should be resolved now, or could have an RFC Editor note added indicating
that the author will resolve that during Authors 48 hours.

Nits/editorial comments:

idnits didn't like the reference to RFC 20 for ASCII:

  ** Downref: Normative reference to an Unknown state RFC: RFC   20

RFC 5234 (ABNF) uses this, which looks like a better reference:

   [US-ASCII]  American National Standards Institute, "Coded Character
               Set -- 7-bit American Standard Code for Information
               Interchange", ANSI X3.4, 1986.

--- Selected RFC 5706 Appendix A Q&A for OPS-Dir review ---

Most of these questions are n/a because this draft describes a format
that will be used in other protocols to which RFC 5706's concerns would apply.

A.1.4   Have the Requirements on other protocols and functional
       components been discussed?

The specification of the interaction of the JSON sequence parser with the
JSON text parser is not as clear as it should be for incomplete or malformed
JSON texts.  See Minor Issues [A]-[E] above.

A.1.8   Are there fault or threshold conditions that should be reported?

Yes, incomplete JSON texts - this is covered in sections 2.3 and 2.4. 

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Barry Leiba <[email-masked]>
Date: Fri, 5 Dec 2014 17:38:54 -0500

>                Interchange", ANSI X3.4, 1986.

Except that (1) many IETF documents do use RFC 20 and (2) the RFC 20
reference is not for ASCII: it's for RS, the Record Separator
character, which is explained in RFC 20, Section 5.2.

Barry

---

From: "Black, David" <[email-masked]>
Date: Fri, 5 Dec 2014 22:54:04 +0000

Ok, it is a nit, as is much of what idnits produces ;-).
I have no problem ignoring idnits on this one ...

Thanks,
--David

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Barry Leiba
> Sent: Friday, December 05, 2014 5:39 PM
> To: Black, David
> Cc: [email-masked]; General Area Review Team ([email-masked]); ops-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Gen-ART and OPS-Dir review of draft-ietf-json-text-sequence-09
>

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 7 Dec 2014 10:40:43 +0100

All,

Many definitions in this document have been specified in the form of 1:1 mappings of Unicode code points to single bytes. This include the record separator and for example definitions of the string "false" in the form of five octets.

This is ok if the encoding used for Unicode is UTF-8, and indeed it says that the parser should try to parse the string as if it is a UTF-8 sequence of characters.

But it also reference RFC7159, which doesn't require UTF-8 but instead for some weird reason also allow other encodings of Unicode text. And on top of that it says Byte Order Mark is not allowed.

This together implies that first of all this draft might not lead to stable implementations, secondly one can not store in JSON strings that include the Byte Order Mark, and there are other unspecified situations.

Or, in short, weaknesses, as I see it, in RFC7159 are made even more weak and potentially dangerous with draft-ietf-json-text-sequence.

Yes, I and others should probably not have let RFC7159 through, because there might be where the bugs are.

Suggestion: this draft, draft-ietf-json-text-sequence, should say explicitly that the only "profile" of RFC7159 that is allowed is UTF-8. That should be a MUST.

Reminder for the IETF: having "or" statements is not recommended at all when talking about these kind of things, and RFC7159 include at least one "or" too many. The recommendation from IETF is to use UTF-8 encoding for Unicode (when serializing text).

   Patrik

> On 5 dec 2014, at 15:51, Black, David <[email-masked]> wrote:
>

---

From: Pete Resnick <[email-masked]>
Date: Sun, 7 Dec 2014 10:41:00 -0600

> character, which is explained in RFC 20, Section 5.2.

And as per BCP97, RFC 3967, section 3, this downref was specifically 
called out in the Last Call for this document.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Dec 2014 13:05:28 -0500

Patrik Fältström scripsit:

> But it also reference RFC7159, which doesn't require UTF-8 but instead
> for some weird reason also allow other encodings of Unicode text. And
> on top of that it says Byte Order Mark is not allowed.

7159 was meant to tighten the wording of 4627, not to impose additional
constraints on it.  For that, see the I-JSON draft.

> This together implies that first of all this draft might not lead to
> stable implementations, secondly one can not store in JSON strings
> that include the Byte Order Mark, and there are other unspecified
> situations.

If by that you mean that a JSON string may not contain U+FEFF, that is
incorrect, for U+FEFF is recognized as a BOM only when placed at the
beginning of an entity body, whereas an entity body in JSON format can
begin only with [ or { classically, or by extension with [0-9"tfn].

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Micropayment advocates mistakenly believe that efficient allocation of
resources is the purpose of markets.  Efficiency is a byproduct of market
systems, not their goal.  The reasons markets work are not because users
have embraced efficiency but because markets are the best place to allow
users to maximize their preferences, and very often their preferences are
not for conservation of cheap resources.  --Clay Shirky

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 7 Dec 2014 19:55:41 +0100

> On 7 dec 2014, at 19:05, John Cowan <[email-masked]> wrote:
> 

The problem I have is that 7159 is not tight enough as it allows other encodings than UTF-8, which in turn make the encoding not work very well as this draft take for granted each one of the separator characters is one byte each.

I.e. the way I read draft-ietf-json-text-sequence (and I might be wrong), you have specific octet values that act as separators. That only works if the encoding is UTF-8.

See Figure 1:

> possible-JSON = 1*(not-RS); attempt to parse as UTF-8-encoded

Now, if this is NOT UTF-8, then this might be pretty bad situation.

What I am saying is that I would like this draft to explicitly say that the only profile of RFC7159 that can be used is when UTF-8 is in use, i.e. somewhere something like "The encoding MUST be UTF-8, although RFC7159 also allow other encodings, like UTF-16." Then in the security considerations section that "RFC7159 do allow not only UTF-8 encoding but also for example UTF-16, which MIGHT create problems for a parser, all depending on what data is serialized."

I.e. I want this draft to be even more tight than RFC7159.

Let me ask it this way: is there any reason to allow other encodings than UTF-8? If so, how do you handle the encoding of the separators?

> begin only with [ or { classically, or by extension with [0-9"tfn].

Ok, so what you say is that a string in an attribute value in the JSON blob can still start with U+FEFF?

If so, good, and my apologies for not understanding this at my read of the text.

   Patrik

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Dec 2014 16:07:54 -0500

Patrik Fältström scripsit:

> I.e. the way I read draft-ietf-json-text-sequence (and I might be
> wrong), you have specific octet values that act as separators. That
> only works if the encoding is UTF-8.

This is a binary representation which has embedded JSON texts represented
in UTF-8.  Since the first character in a JSON text is necessarily in
the ASCII repertoire, it is not possible to parse a UTF-16 or UTF-32
JSON text as UTF-8 and come out with valid JSON.

However, I grant that mentioning UTF-8 only in an ABNF comment is not
really prominent enough.  Proposed wording change:

For:

   In prose: a series of octet strings, each containing any octet other
   than a record separator (RS) (0x1E) [RFC0020], all octet strings
   separated from each other by RS octets.  Each octet string in the
   sequence is to be parsed as a JSON text.

read:

   In prose: a series of octet strings, each containing any octet other
   than a record separator (RS) (0x1E) [RFC0020], all octet strings
   separated from each other by RS octets.  Each octet string in the
   sequence is to be parsed as a JSON text in UTF-8 encoding.

and add a suitable reference to UTF-8.

> blob can still start with U+FEFF?

Just so.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
As we all know, civil libertarians are not the friskiest group around --
comes from  forever being on the qui vive for the sound of jack-booted
fascism coming down the pike.           --Molly Ivins

---

From: "Black, David" <[email-masked]>
Date: Mon, 8 Dec 2014 02:59:44 +0000

I agree with Patrik - this draft assumes UTF-8 encoding and should
state that requirement explicitly.  John's proposed text change below
is in section 2.1 for the decoder; the encoder text in section 2.2
needs a corresponding change:

OLD
   In prose: any number of JSON texts, each preceded by one ASCII RS
   character and each followed by a line feed (LF).
NEW
   In prose: any number of JSON texts encoded as UTF-8, each preceded
   by one ASCII RS character and each followed by a line feed (LF).

Thanks,
--David

> From: John Cowan [mailto:[email-masked]] On Behalf Of John Cowan
> Sent: Sunday, December 07, 2014 4:08 PM
> To: Patrik Fältström
> Cc: Black, David; Nico Williams; General Area Review Team ([email-masked]);
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [Json] Gen-ART and OPS-Dir review of draft-ietf-json-text-
> sequence-09
>

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 8 Dec 2014 06:42:32 +0100

> On 7 dec 2014, at 22:07, John Cowan <[email-masked]> wrote:
> 

My point is that if you talk about what specific characters or reference RFC20 or what not, then you only get RS if you use UTF-8 encoding. If you use UTF-16, then you neither have RS as one octet (0x1E), nor is RS the only character that do have 0x1E as one of the octets.

I think the problem is that I do not know what "octet string" is. You either have UTF-8 encoded Unicode strings, or... ;-) In this case, you have a series of UTF-8 encoded Unicode Strings, right? Separated by the octet 0x1E, which happen to also be a correctly encoded Unicode character -- the Information Separator Two. This implies the whole thing is a UTF-8 encoded text that is to be parsed like this:

possible-JSON = 1*(not-RS); UTF-8-encoded JSON text
 ; (as specified in RFC7159, but only UTF-8 allowed)

I.e. the blob, to be compliant with this document, MUST be UTF-8 encoded JSON.

Right?

> and add a suitable reference to UTF-8.

I would say that what you have said above is:

This specifies a series of UTF-8 encoded Unicode strings. Each to be interpreted as JSON text. The strings are separated by the octet 0x1E (which is UTF-8 encoding of the Unicode Character U+001E - INFORMATION SEPARATOR TWO). This character because of this must be escaped, for example by using \u001E notation, if it exists in an attribute value.

>> blob can still start with U+FEFF?

Good.

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 8 Dec 2014 06:44:35 +0100

> On 8 dec 2014, at 03:59, Black, David <[email-masked]> wrote:
> 
> OLD

My point is that you do not have to talk about ASCII RS. You can as well talk about the UTF-8 encoding of the unicode character INFORMATION SEPARATOR TWO, U+001E. Much cleaner.

Just say it must be UTF-8 encoded text, done.

   Patrik

---

From: Tim Bray <[email-masked]>
Date: Sun, 7 Dec 2014 23:37:58 -0800

I agree 100% that the fact that JSON Text Sequences MUST BE UTF-8 should be
highlighted.  Both the ASCII and Unicode names of 30/0x1e/00011110 should,
for completeness, be provided.  It would be inappropriate to omit ASCII,
because we picked 0x1e because of its ASCII name.

On Sun, Dec 7, 2014 at 9:44 PM, Patrik Fältström <[email-masked]> wrote:

> My point is that you do not have to talk about ASCII RS. You can as well
> talk about the UTF-8 encoding of the unicode character INFORMATION
> SEPARATOR TWO, U+001E. Much cleaner.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 8 Dec 2014 08:48:08 +0100

FWIW, I think this is a good view. I just felt that given the other discussion about RFC20, you do not HAVE TO talk about ASCII or the name of the code point in the standard. Sure, that is the reason you picked U+001E, but that is more a nice story ;-)

It was not my intention to say that you should not talk about ASCII.

   Patrik

> On 8 Dec 2014, at 08:37, Tim Bray <[email-masked]> wrote:
> 
> I agree 100% that the fact that JSON Text Sequences MUST BE UTF-8 should be highlighted.  Both the ASCII and Unicode names of 30/0x1e/00011110 should, for completeness, be provided.  It would be inappropriate to omit ASCII, because we picked 0x1e because of its ASCII name.
> 
> - Tim Bray (If you’d like to send me a private message, see https://keybase.io/timbray)

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 21:30:57 -0600

[I've been traveling, please excuse my late responses.]

On Sun, Dec 07, 2014 at 07:55:41PM +0100, Patrik Fältström wrote:
> The problem I have is that 7159 is not tight enough as it allows other
> encodings than UTF-8, which in turn make the encoding not work very
> well as this draft take for granted each one of the separator
> characters is one byte each.
> 

Right.  I'll add text to section 2.2 saying that the JSON texts have to
be encoded in UTF-8.

> Now, if this is NOT UTF-8, then this might be pretty bad situation.

Well, you can always fuzz test a parser... :)

But yes, the encoder should use UTF-8.

> What I am saying is that I would like this draft to explicitly say
> that the only profile of RFC7159 that can be used is when UTF-8 is in
> use, i.e. somewhere something like "The encoding MUST be UTF-8,
> although RFC7159 also allow other encodings, like UTF-16." Then in the
> security considerations section that "RFC7159 do allow not only UTF-8
> encoding but also for example UTF-16, which MIGHT create problems for
> a parser, all depending on what data is serialized."
> 

I agree with this.  This was always my intent (as in: I never intended
to support UTF-16 or UTF-32, say, or any other UTF, in any
implementation of mine).

And I agree that this format doesn't work with UTF-16 or UTF-32 EVEN IF
the UTF were part of the MIME type and UTF-specific multi-byte
separators were used: because at least for log-type applications where
atomicity of writes is in question multi-byte separators won't do.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 21:37:13 -0600

>    sequence is to be parsed as a JSON text in UTF-8 encoding.

Agreed.  And a corresponding change to section 2.2, which will now read:

   In prose: any number of JSON texts, each encoded in UTF-8, each
   preceded by one ASCII RS character, and each followed by a line feed
   (LF). Since RS ...

> and add a suitable reference to UTF-8.

Oh, eh, RFC7159 lacked such a thing.  At least this one should be
non-controversial: RFC3629.  (Right? :)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 21:41:59 -0600

> because we picked 0x1e because of its ASCII name.

Good point.  (It's "Unicode Character 'INFORMATION SEPARATOR TWO' (U+001E)".)

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 22:19:42 -0600

> This is a combined Gen-ART and OPS-Dir review.  Boilerplate for both follows ...

Thanks you.

> of the JSON text that failed to parse.  I suggest:

Good point.

>    the remainder of the sequence from there.

That's a weird way of saying that, wherever the JSON text parse fails,
the sequence parser should then seek forward until the next RS-valued
byte.  But it works for me; I'll take it.

> to parse), and I think that's worth stating.

It's permitted, of course, with the caveat that all incremental parsers
have: the parse can ultimately fail.  I'll add this note:

   Incremental JSON text parsers may be used, though of course failure
   to parse a given text may result after first producing some
   incremental parse results.

to section 2.3.

> instead of the text parse.

I'd wanted to not have to list the characters that are considered
whitespace.  I agree that the "ws" rule is not appropriate though.

> be incomplete because the last character is not LF, e.g.:

A missing terminating LF is not a problem for strings, arrays, or
objects.  I seem to recall that we did discuss this.  We could require
that such texts fail to parse, but perhaps the more important thing is
to require common parser behavior as to such truncations.

You ABNF proposal is certainly more strict than the one in the I-D.  I'm
neutral as to whether this form or the one in the I-D (with the ws issue
fixed) is better.  The stricter form is clearly easier to talk about,
therefore preferable, but it will mean discarding texts where only that
terminating LF is truncated.

One problem we get into with your ABNF is that RFC5234 does not discuss
greediness (this came up on the list).  But this can be noted (see
below).

One nice thing about the stricter rules is that we need not discuss
top-level numbers (or booleans, or null) with normative text, just note
them.

> within a JSON text (and this should be stated).

It will if ABNF matching is greedy and possible-JSON is defined as

   possible-JSON = 1*( 1*(not-RS) LF); ...

Advice as to which form to take?

> considerations and incomplete texts are a relevant consideration for

And surely also for RFC7159.  Besides requiring that they fail to parse
(and the note about incremental parsing), are there any other missing
considerations?  Ah yes, smuggling of data in sequences where parsers
will ignore without warning any octet strings that fail to parse as JSON
texts.

Proposed text:

   As usual, parsers must operate on as-good-as untrusted input. This
   means that parsers must fail gracefully in the face of malicious
   inputs. Note that incremental parsers can produce partial results and
   later indicate failure to parse the remainder of a text. Note that
   texts that fail to parse and are ignored can be used to smuggle data
   past sequence parsers that don't warn about JSON text failures.

> that the author will resolve that during Authors 48 hours.

Sure.

>   ** Downref: Normative reference to an Unknown state RFC: RFC   20

Is this resolved by now?  I can always reference only Unicode.

Thanks for your excellent review,

Nico
--

---

From: John Cowan <[email-masked]>
Date: Mon, 8 Dec 2014 23:20:25 -0500

Patrik Fältström scripsit:

> This implies the whole thing is a UTF-8 encoded text that is to be
> parsed like this:

No, this is a misunderstanding.  There is no requirement that the sequence
*as a whole* is well-formed UTF-8 text.  For example, if the first JSON
text is written only in part for whatever reason (system crash, etc.) to
a log file, the next process can write a 0x1E byte and carry on.

> I.e. the blob, to be compliant with this document, MUST be UTF-8 encoded JSON.

The possible-JSON part is either UTF-8 encoded JSON or it's arbitrary junk.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Said Agatha Christie / To E. Philips Oppenheim
"Who is this Hemingway? / Who is this Proust?
Who is this Vladimir / Whatchamacallum,
This neopostrealist / Rabble?" she groused.
        --George Starbuck, Pith and Vinegar

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 22:27:13 -0600

> a log file, the next process can write a 0x1E byte and carry on.

Correct.  But we're splitting hairs: arbitrary octet strings (but for
0x1E) can be attempted to be parsed, though only those that are valid
JSON texts encoded in UTF-8 should be accepted.

The encoder, of course, must produce UTF-8.

Nico
--

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 9 Dec 2014 06:03:33 +0100

> On 9 dec 2014, at 04:37, Nico Williams <[email-masked]> wrote:
> 

Yes, correct.

   Patrik

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 23:11:37 -0600

> > non-controversial: RFC3629.  (Right? :)

I received one off-list response that RFC3629 would be inappropriate
because Unicode is a versioned standard.  But here we want a UTF, not
all of Unicode, and surely UTF-8 is stable.

Nico
--

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 09 Dec 2014 16:08:50 +0900

> all of Unicode, and surely UTF-8 is stable.

Yes indeed.    Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 9 Dec 2014 08:32:15 +0100

> On 9 dec 2014, at 05:20, John Cowan <[email-masked]> wrote:
> 

Ok, fair.

Then I think the draft itself should state this in one way or another. If nothing else the text you wrote here or a version of it is to go into the security considerations section.

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 9 Dec 2014 13:46:36 +0100

> On 9 dec 2014, at 06:11, Nico Williams <[email-masked]> wrote:
> 

RFC3629 has no limitation based on Unicode version. The limitation is that it only defines UTF-8 for the UTF-16 accessible range, U+0000..U+10FFFF.

   Patrik

---

From: "Black, David" <[email-masked]>
Date: Tue, 9 Dec 2014 16:41:12 +0000

Nico,

Thanks for the comprehensive response.

I've excerpted a few things for further comment - I'm fine with the
proposed actions for anything not mentioned here.

[A] JSON text parse failures

> byte.  But it works for me; I'll take it.

Your alternative wording "whenever the JSON text parse fails, ..." is fine.

[D] Truncation

> terminating LF is truncated.

I concur with both of the above paragraphs - my preference is to detect
incomplete JSON texts at the sequence level via the missing LF rather than
special-casing numbers and relying on failed JSON parses for everything else.
In general, earlier detection of errors increases the options for dealing
with them.

Once the incomplete text is detected, a JSON parse could be attempted,
with the JSON parser knowing that the text is incomplete (e.g., text
may fail to parse, a number at the end of the text must not be produced
as an incremental parse result).

I'll defer to the WG on whether/how to expand the decoder ABNF to better
detect incomplete texts and how to deal with any incomplete texts that are
detected.

As for RFC 20 ...

> Is this resolved by now?  I can always reference only Unicode.

Keep the RFC 20 reference - I have no problem with it.  Moreover, as a
result of all the hubbub around this nit, the IESG has issued a Last Call
to reclassify RFC 20 as an Internet Standard ... so that this never
arises again ...

http://www.ietf.org/mail-archive/web/ietf-announce/current/msg13546.html

Thanks,
--David

> From: Nico Williams [mailto:[email-masked]]
> Sent: Monday, December 08, 2014 11:20 PM
> To: Black, David
> Cc: General Area Review Team ([email-masked]); [email-masked];
> [email-masked]; [email-masked]

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 11:17:29 -0600

On Tue, Dec 09, 2014 at 04:41:12PM +0000, Black, David wrote:
> > [...]

OK.

> In general, earlier detection of errors increases the options for dealing

And, of course, a streaming/incremental parsers might well output all
there is to output when only the last LF is missing but the top-level
value was properly delimited anyways.  So it's kinda difficult to get a
fool-proof requirement that the trailing LF must be present.

Your review comments included adding this note about incremental
parsing.  There's a conflict here between the two comments that had not
been apparent to me last night.  I now think that fixing the ws problem
is the best way forward.

> as an incremental parse result).

That's so for non-incremental parsers.  (Or when buffering the complete
text instead of handling incrementally, even though one has an
incremental parser.)

Consider one implementation I'm familiar with.  Its JSON text parser is
incremental (but not streaming), so it produces outputs with no need for
extra whitespace when the input text is a string, array, or object, but
for top-level numbers, booleans, and null, it needs to either read one
more byte or reach EOF before it will output them.

So I think we really do need to say something about top-level numbers
(and true, false, and null), namely: that they must be delimited by
whitespace, that '<RS>1234<RS>' is not a valid sequence element because
the number may have been truncated.  (Ditto '<RS>true<RS>', since the
intended text could have been 'trueish', which is invalid of course, but
still.)

> to reclassify RFC 20 as an Internet Standard ... so that this never

Yes, I noticed.  I expect the IETF LC will pass for that.

Thanks,

Nico
--

---

From: "Black, David" <[email-masked]>
Date: Tue, 9 Dec 2014 18:49:35 +0000

> intended text could have been 'trueish', which is invalid of course, but

That would be more robust, as then all JSON texts in a sequence have
delimiters and absence of the closing delimiter clearly indicates
truncation.

Thanks,
--David

> Sent: Tuesday, December 09, 2014 12:17 PM
> To: Black, David

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 18:49:30 -0600

OK.

New section 2.4 text:

   While objects, arrays, and strings are self-delimited in JSON texts,
   numbers, and the values 'true', 'false', and 'null' are not.  Only
   whitespace can delimit the latter four kinds of values.

   Parsers MUST check that any JSON texts that are a top-level number,
   or which might be 'true', 'false', or 'null' include JSON whitespace
   (at least one byte matching the "ws" ABNF rule from RFC7159) after
   that value, otherwise the JSON-text may have been truncated.  Note
   that the LF following each JSON text matches the "ws" ABNF rule.

   Parsers MUST drop JSON-text sequence elements consisting of
   non-self-delimited top-level values that may have been truncated
   (that are not delimited by whitespace).  Parsers can report such
   texts as warnings (including, optionally, the parsed text and/or the
   original octet string).

   For example, '<RS>123<RS>' might have been intended to carry the
   top-level number 123.4, but must have been truncated.  Similarly,
   '<RS>true<RS>' might have been intended to carry the invalid text
   'trueish'.  '<RS>truefales<RS>' is not two top-level values, 'true',
   and 'false'; it is simply not a valid JSON text.

This is the only place where the ws rule comes up, so merely saying "at
least one byte matching" it should suffice.

I'm also adding this following the above, based on your comment about
incremental parsers:

   Implementations may produce a value when parsing '<RS>"foo"<RS>'
   because their JSON text parser might be able to consume bytes
   incrementally, and since the JSON text in this case is a
   self-delimiting top-level value, the parser can produce the result
   without consuming an additional byte.  Such implementations should
   skip to the next RS byte, possibly reporting any intervening
   non-whitespace bytes.

(yes, I think this should be a 'should', not a 'SHOULD').

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 19:06:13 -0600

On Wed, Dec 10, 2014 at 10:52:22AM +1000, Matthew Kerwin wrote:
> On 10 December 2014 at 10:49, Nico Williams <[email-masked]> wrote:
> Not an 'ought to'?

I'll take that.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 10 Dec 2014 10:52:22 +1000

> (yes, I think this should be a 'should', not a 'SHOULD').
​Not an 'ought to'?​

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: "Black, David" <[email-masked]>
Date: Wed, 10 Dec 2014 02:52:34 +0000

This looks good, and the explanation of what's going on (numbers,
'true', 'false' and 'null' lack delimiters) is a useful addition.

One small nit:

	'<RS>truefales<RS>' is not two top-level values, 'true',
	and 'false'; it is simply not a valid JSON text.

truefales -> truefalse

Thanks,
--David

---

From: John Cowan <[email-masked]>
Date: Wed, 10 Dec 2014 20:31:37 -0500

Patrik Fältström scripsit:

> RFC3629 has no limitation based on Unicode version. The limitation
> is that it only defines UTF-8 for the UTF-16 accessible range,
> U+0000..U+10FFFF.

That is the complete Unicode range.  There are no Unicode scalar values
above U+10FFFF and never will be (as distinct from Unicode scalar values
that are not, or not yet, assigned to characters).  So that is no
limitation at all.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Let's face it: software is crap. Feature-laden and bloated, written under
tremendous time-pressure, often by incapable coders, using dangerous
languages and inadequate tools, trying to connect to heaps of broken or
obsolete protocols, implemented equally insufficiently, running on
unpredictable hardware -- we are all more than used to brokenness.
                   --Felix Winkelmann