Subject: [Json] Leading and trailing whitespace

From: "Jim Schaad" <[email-masked]>
Date: Mon, 10 Jun 2013 14:54:19 -0700

The current specification allows for arbitrary whitespace to occur before
and after an array or object.  I would like to know if this is what was
intended to begin with or not.

There are two different things that could be done about this (one of which
would potentially be necessary if you allowed all values in a JSON-text.

1.  Some of the ABNF could be cleaned up to eliminate situations where you
end up with two ws non-terminals in a row.  Thus the ABNF could be modified
from

   JSON-text = object / array

      begin-array     = ws %x5B ws  ; [ left square bracket

      begin-object    = ws %x7B ws  ; { left curly bracket

      end-array       = ws %x5D ws  ; ] right square bracket

      end-object      = ws %x7D ws  ; } right curly bracket

      name-separator  = ws %x3A ws  ; : colon

      value-separator = ws %x2C ws  ; , comma

      object = begin-object [ member *( value-separator member ) ]
               end-object

      member = string name-separator value

   array = begin-array [ value *( value-separator value ) ] end-array

to

JSON-text = object / array
begin-array = %x5B  ws ; [ left square bracket
begin-object = %x7B ws ; { left curly brace
end-array = ws %5D ; ] right square bracket
end-object = ws %7D ; } right curly brace
name-separator = ws %x3A ws ; : colon
value-separator = ws %x2C ws ; , comma

If needed then JSON-text would become either
JSON-text = ws value ws 
Or
JSON-text = ws object ws / ws array ws

2.  There is nothing in the current parser or generator text which says what
to do if you come across a string which either has leading/trailing white
space or a string which has a complete JSON-text string followed by
non-white space characters.   I would hope that the second case should cause
either an error or some other indicator that the entire string was not
consumed.  I am slightly agnostic about the first case but would kind of
like to eliminate leading and trailing whitespace.

Jim

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 15:04:15 -0700

<no hat>

> intended to begin with or not.

Why is the "intention" important?

> would potentially be necessary if you allowed all values in a JSON-text.

Why should something "be done about this"? I think you are leading into canonicalization, but that's not part of RFC 4627, so introducing it seems like a pretty massive change.

Having said that, knowing your intention would be useful here.

--Paul Hoffman

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 10 Jun 2013 17:16:24 -0700

My intention here has nothing to do with canonicalization and everything to
do with what constitutes a valid string from a parsing perspective.

Consider for example the case of a text file which consists of a JSON text
string with a trailing CRLF in the file.  If trailing whitespace is allowed
then the entire text file is a legal JSON text string.  If the trailing
whitespace is not allowed then it is not a legal JSON text string.  I am
just trying to clarify which is true.

This makes a difference with the argument about moving all of the values to
the top level since then the string (ignore the quotes) "   true \r\n"
cannot be matched if you just want to say it is a Boolean literal (i.e. the
non-terminator true).  Instead you need to say that it can have whitespace
on either side of it.

I will agree that the original intention probably is just a question of
academic interest.  The real question I should have asked would be are
people taking advantage of the fact.

That is the question of what about normal whitespace.

I think there is a question about parsers that would accept the string

{"a":"b"}A

As I believe there are some that will (stop when you get to the end of the
object) and some that will not.

Jim

> -----Original Message-----
> From: Paul Hoffman [mailto:[email-masked]]
> Sent: Monday, June 10, 2013 3:04 PM
> To: Jim Schaad
> Cc: [email-masked]
> Subject: Re: [Json] Leading and trailing whitespace
> 
> <no hat>
> 
> > The current specification allows for arbitrary whitespace to occur
> > before and after an array or object.  I would like to know if this is
> > what was intended to begin with or not.
> 
> > There are two different things that could be done about this (one of
> > which would potentially be necessary if you allowed all values in a
JSON-
> text.
> 
> Why should something "be done about this"? I think you are leading into
> canonicalization, but that's not part of RFC 4627, so introducing it seems
like a
> pretty massive change.
> --Paul Hoffman=

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 19:01:41 -0700

> just trying to clarify which is true.

The "JSON text string" (called a "string" in RFC 4627) starts and ends with a quotation mark:
      string = quotation-mark *char quotation-mark
There is no "ws" anywhere there, but there sure are quotation marks.

> non-terminator true).  Instead you need to say that it can have whitespace

The current grammar does not allow whitespace around "true" for that boolean value.

> That is the question of what about normal whitespace.

I would be interested if anyone sees anything in the document that disagrees with the above.

> object) and some that will not.

If that is meant to be a JSON text, I cannot imagine how that is valid, but maybe I'm not creative enough.

--Paul Hoffman

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 10 Jun 2013 19:32:19 -0700

> Sent: Monday, June 10, 2013 7:02 PM
> To: Jim Schaad
> Cc: [email-masked]
> > My intention here has nothing to do with canonicalization and
> > everything to do with what constitutes a valid string from a parsing
> perspective.
> >
> > Consider for example the case of a text file which consists of a JSON
> > text string with a trailing CRLF in the file.  If trailing whitespace
> > is allowed then the entire text file is a legal JSON text string.  If
> > the trailing whitespace is not allowed then it is not a legal JSON
> > text string.  I am just trying to clarify which is true.
> 
> The "JSON text string" (called a "string" in RFC 4627) starts and ends
with a
> quotation mark:
>       string = quotation-mark *char quotation-mark There is no "ws"
anywhere
> there, but there sure are quotation marks.
> 

Ok - I used the wrong term - I need to get them clearer, I still am trying
to learn - Please read JSON text for JSON text string.  I was not using
string as a value in this location.

> > cannot be matched if you just want to say it is a Boolean literal
> > (i.e. the non-terminator true).  Instead you need to say that it can
> > have whitespace on either side of it.
> 

Actually it does, but the whitespace comes from the container it is in.  The
current grammar does not allow true at the top level and if that does not
change then the above remains a moot point.  Note that the text was talking
about moving all values to the top level.

> 
> > I will agree that the original intention probably is just a question
> > of academic interest.  The real question I should have asked would be
> > are people taking advantage of the fact.
> >
> If that is meant to be a JSON text, I cannot imagine how that is valid,
but
> maybe I'm not creative enough.

Yes -but what I am saying is that there are (probably) parsers which stop
before the string has been fully consumed.  Thus they would accept the
string as valid.

Jim

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 10 Jun 2013 19:53:25 -0700

> string as a value in this location.

You can't ask that question because the current doc only allows arrays and objects at the top level, and the starting character for each of those allow for white space.

You might be asking "if we allow other JSON items at the top level, will we also allow optional whitespace before and after". But that's a supposition, not a question about what is in the document now.

> Actually it does, but the whitespace comes from the container it is in.  

You have changed the subject of the question, then. You were asking about the top level, without a container. I claim my answer is correct for the question you asked.

> The

None of the objects other than arrays and objects have optional whitespace. None.

> before the string has been fully consumed.  Thus they would accept the

There are (probably) parsers that do all sorts of stupid, non-conformant things. Why is that relevant here?

--Paul Hoffman

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 10 Jun 2013 20:01:33 -0700

> Sent: Monday, June 10, 2013 7:53 PM
> To: Jim Schaad
> Cc: [email-masked]
> > Ok - I used the wrong term - I need to get them clearer, I still am
> > trying to learn - Please read JSON text for JSON text string.  I was
> > not using string as a value in this location.
> 
> You can't ask that question because the current doc only allows arrays and
> objects at the top level, and the starting character for each of those
allow for
> white space.
> 
> You might be asking "if we allow other JSON items at the top level, will
we
> also allow optional whitespace before and after". But that's a
supposition,
> not a question about what is in the document now.

That must mean I still don't have the correct term

I have not changed the subject here.  

{"a":"b"}\r\n

Is that legal and should it be legal - the current ABNF says yes.  What do
current parsers say?

> 
> 
> >>> This makes a difference with the argument about moving all of the
> >>> values
> >> to
> You have changed the subject of the question, then. You were asking about
> the top level, without a container. I claim my answer is correct for the
> question you asked.
> 
> > The
> > current grammar does not allow true at the top level and if that does
> > not change then the above remains a moot point.  Note that the text
> > was talking about moving all values to the top level.
> 
> None of the objects other than arrays and objects have optional
whitespace.
> None.
> 
> > Yes -but what I am saying is that there are (probably) parsers which
> > stop before the string has been fully consumed.  Thus they would
> > accept the string as valid.
> 

We made some explicit decisions in the JOSE working group based on some of
the above issues.  I was just verifying that we had the correct assumptions
in making those decisions.

---

From: Vinny A <[email-masked]>
Date: Mon, 10 Jun 2013 22:07:11 -0500

> before the string has been fully consumed.  Thus they would accept the

There may be parsers that stop parsing at the end }, but (IMO) that sounds like a lazy parser. I just tested the Golang encoding/JSON default package and it rejects your JSON as invalid.

It sounds to me that you're trying to make the larger point of "what text, if any, is valid outside of the JSON braces." Whitespace - to me - is syntactic sugar. Even leading and trailing whitespace is helpful: it provides space for future comments, for example.

-----------------
Vinny
www.jsontest.com

---

From: Nico Williams <[email-masked]>
Date: Mon, 10 Jun 2013 22:42:21 -0500

> You might be asking "if we allow other JSON items at the top level, will we also allow optional whitespace before and after". But that's a supposition, not a question about what is in the document…

Sure, but since we're full-out having the any value type at the
top-level discussion (and it looks like we'll at least have to have
informational text acknowledging that ECMAScript will allow them) this
becomes relevant.

Do implementations that allow any value type at the top-level allow
leading and/or trailing whitespace?

*Should* they?

IMO the intuitive answer to the last should be "yes", that arbitrary
amounts of whitespace can surround any value regardless of where it
appears.  But my intuition could be wrong.

> You have changed the subject of the question, then. You were asking about the top level, without a container. I claim my answer is correct for the question you asked.

Well, that's what happens when people have discussions...  :)

> None of the objects other than arrays and objects have optional whitespace. None.

Indeed.  But that might need to change with the top-level value type
discussion.  Or if not "need", we might want it to anyways.

> There are (probably) parsers that do all sorts of stupid, non-conformant things. Why is that relevant here?

Actually, that seems very relevant for the Security Considerations
section.  What should parsers do with any content left over after
parsing the end of a top-level value?

Trailing whitespace would allow parsers to delimit top-level numeric
values when end-of-message might be hard to distinguish from a TCP
RST.

Trailing content might allow smuggling of data past a JSON parser,
where it could do damage (e.g., get past deep packet inspectors).

Perhaps the simplest thing to do would be to allow just one space to
trail top-level values.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Jun 2013 07:14:25 +0200

> leading and/or trailing whitespace?

Yes.  E.g.,

>> JSON.parse("\r\n \t 27 \r\n \t", quirks_mode: true)
=> 27

(quirks_mode is the flag that switches on the non-coforming acceptance of any "value".)

> *Should* they?

Yes.

(If you are not convinced from first principles: people have argued you can do streaming by adding whitespace, and that required being allowed to add whitespace :-)

So we will have to add a production:

	JSON-value = ws value ws

That didn't seem too hard.

Thanks for pointing this out, Jim.

This production can be in 4627-bis or in the document that defines the application/json-value media type, if we want to go the latter route.

Grüße, Carsten

PS.: Just to show that trailing whitespace is indeed properly parsed:
>> JSON.parse("\r\n \t 27 \r\n \t 28", quirks_mode: true)
JSON::ParserError: 795: unexpected token at '28'

---

From: Nico Williams <[email-masked]>
Date: Tue, 11 Jun 2013 01:08:43 -0500

> (quirks_mode is the flag that switches on the non-coforming acceptance of any "value".)

Right.  I suppose I meant "by default" :)

> (If you are not convinced from first principles: people have argued you can do streaming by adding whitespace, and that required being allowed to add whitespace :-)

Indeed, I've argued as much.  The whitespace is only needed to
distinguish end-of-message from a pemature EOF that might not
otherwise be understood as such.

> That didn't seem too hard.

Do we need to remove whitespace elsewhere in the ABNF?  Or is
redundancy there OK?

> Thanks for pointing this out, Jim.

Yes, thanks!

> This production can be in 4627-bis or in the document that defines the application/json-value media type, if we want to go the latter route.

Sure.  For now I'm still arguing for allowing any value type at the top-level.

> JSON::ParserError: 795: unexpected token at '28'

Ah, good job.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Jun 2013 10:56:25 +0200

> Right.  I suppose I meant "by default" :)

Without quirks mode, the Ruby JSON parser only accepts JSON-texts (i.e., maps and arrays), 
so I couldn't have tested how whitespace is accepted around top-level values.

> Do we need to remove whitespace elsewhere in the ABNF?  Or is

Redundant opportunities for whitespace in the ABNF are not a problem.

Grüße, Carsten

---

From: Stephan Beal <[email-masked]>
Date: Tue, 11 Jun 2013 12:10:26 +0200

string as valid.

All of the parsers i've written or worked closely with stop processing at
the top of the closing tag for the root element. It doesn't generically
make sense to keep reading from a stream when you have received the virtual
EOF byte (the closing tag).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 11 Jun 2013 09:06:11 -0700

> That didn't seem too hard.

Question: are you saying we should do that even if we don't allow all values at the top level, or only if we add them at the top level?

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Jun 2013 18:16:40 +0200

> Question: are you saying we should do that even if we don't allow all values at the top level, or only if we add them at the top level?

This production does not have any effect unless we use it.
The radical proposal is to replace the top-level JSON-text with it, or

JSON-text = ws value ws

My less radical, but somewhat questionable proposal is to use it as a second "top-level", i.e., keep 
JSON-text as is as the basis for application/json and the +json media types, and add application/json-value referencing this new production.

The less radical proposal is somewhat questionable because it is very likely that people will keep sending around JSON-value representations with a application/json media type even it is clarified that this one is JSON-text only.  On the other hand I haven't seen too many documented cases of that, so maybe it is not that much of a problem.  Clearly, some thinking would need to go into how +json would handle the desire for isolated atomic JSON values.  And, some phrases as in "send it to me in JSON" would need some recalibration, because it is no longer clear which variant is needed.  (Arguably, that hasn't really been clear for a while.)

Grüße, Carsten

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 11 Jun 2013 10:29:08 -0700

On Mon, Jun 10, 2013 at 8:42 PM, Nico Williams <[email-masked]>wrote:

> we also allow optional whitespace before and after". But that's a
> supposition, not a question about what is in the document now.
>
Yes. I was surprised to even see the question -- since white space is
allowed between array and object entries, it would seem surprising if
whitespace at root context behaved differently.
At least I have never encountered a case where parser discarded piece of
JSON due to root-level whitespace (I mostly work on java)

The most practical reason is really that when humans edit JSON, trailing
space after root object or array may be added by text editor. White spce is
all about readability.

> appears.  But my intuition could be wrong.

I think it should be allowed unless there are actual clear reasons not to
allow it.
And unlike with XML limitation for no whitespace before XML declaration,
there does not seem to be solid reasons, other than perhaps suggestion that
since this was omitted from definition it should not be added.

-+ Tatu +-

---

From: Jacob Davies <[email-masked]>
Date: Tue, 11 Jun 2013 10:56:27 -0700

> since this was omitted from definition it should not be added.

It's clearly allowed in the RFC:

object = begin-object [ member *( value-separator member ) ] end-object

begin-object    = ws %x7B ws

end-object      = ws %x7D ws

begin-array     = ws %x5B ws

end-array       = ws %x5D ws

ws = *( %x20 / %x09 / %x0A / %x0D )

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 11 Jun 2013 11:30:14 -0700

> ws = *( %x20 / %x09 / %x0A / %x0D )

Even better -- I was assuming there was some confusion, based on discussion.
Apparently not.

-+ Tatu +-