Subject: [Json] "Generators SHOULD escape all Unicode whitespace characters"?

From: Jacob Davies <[email-masked]>
Date: Mon, 10 Jun 2013 15:54:41 -0700

I'm curious if anyone else thinks this is worth suggesting to
implementors. There are a number of non-ASCII Unicode whitespace and
control characters that are not required to be escaped right now.

I think generators SHOULD escape them. Obviously parsers must continue
to accept them unescaped regardless. The set is fairly small and could
be enumerated in the document (it might expand in future, but this
would be a good start):

http://en.wikipedia.org/wiki/Space_(punctuation)#Spaces_in_Unicode

"Whitespace smuggling" is a mild security concern and, from
experience, can be quite hard to debug if non-0x20 spaces are not
escaped. There is a small overhead of a couple of characters in doing
so.

Everything else in JSON's text serialization uses either printing
characters, insignificant ASCII whitespace between values, or plain
spaces in strings. Of course some printing Unicode characters are
doppelgängers so perhaps people feel it is not worth worrying about
whitespace either.

---

From: Stephen Dolan <[email-masked]>
Date: Tue, 11 Jun 2013 11:58:10 +0100

On Mon, Jun 10, 2013 at 11:54 PM, Jacob Davies <[email-masked]> wrote:
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

There is an interesting issue with unescaped unicode whitespace:
ECMAScript does not allow U+2028 LINE SEPARATOR inside a string
literal, yet JSON does (as it is not one of the listed
must-always-be-escaped characters).

See http://timelessrepo.com/json-isnt-a-javascript-subset

Stephen

---

From: Norbert Lindenberg <[email-masked]>
Date: Tue, 11 Jun 2013 21:35:40 -0700

> http://en.wikipedia.org/wiki/Space_(punctuation)#Spaces_in_Unicode

This list includes some but not all Unicode control characters in addition to space characters.

> escaped. There is a small overhead of a couple of characters in doing

Can you provide more detail on the problem that this proposal is intended to solve? Does the proposal really solve the problem, given that generators don't have to implement it, that they cannot implement it for characters added to Unicode in a Unicode version later than the one they're based on, and that parsers cannot rely on generators to have implemented it?

Norbert

---

From: Jacob Davies <[email-masked]>
Date: Thu, 13 Jun 2013 16:47:47 -0700

> This list includes some but not all Unicode control characters in addition

Yes, and languages vary in what they actually consider "whitespace". I
think in general we're concerned with "non-printing or whitespace
characters other than a simple space".

> Can you provide more detail on the problem that this proposal is intended
> to solve?

Sure - what sometimes happens is that various parts of a system disagree
over what is whitespace. For instance, a server may strip whitespace using
Java's built-in check that does not recognize all of the above-mentioned
characters -
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Character.html#isWhitespace(int)-
when the intent was to remove all whitespace. Malicious users may use
characters that evade this check to introduce whitespace to content in ways
that are unwanted or misleading. In other cases some system may tokenize by
whitespace but varying definitions of what whitespace is result in
different tokenizations and security concerns (as in the case Stephen Dolan
mentions earlier).

It may also assist in debugging seemingly-identical JSON strings that
differ only by invisible or indiscernible whitespace, whether malicious or
intentional.

Does the proposal really solve the problem, given that generators don't
> have to implement it, that they cannot implement it for characters added to
> Unicode in a Unicode version later than the one they're based on, and that
> parsers cannot rely on generators to have implemented it?
>

It certainly does not solve it. It mitigates it in the same way that
escaping control characters and ASCII whitespace in strings mitigate
similar concerns; they make it easier to see exactly what a string is
intended to contain, in human-readable characters.

The case it helps mitigate is the common one where a non-malicious
generator is sending the JSON you're trying to understand, as for instance
when a site's Javascript is communicating with its own server. One of the
nice things about JSON is that it is easy to debug problems in JSON data
using primitive tools - dumping text into page content, or hitting a URL
directly and looking at the JSON in the browser. As much as possible,
implementations should assist.

The recommendation could list a specific set of current characters and
additionally refer to the whitespace and control characters in the latest
Unicode version. As a mitigation measure it helps even though it is partial.

This may be a candidate for a best practice recommendation instead; I
thought it was worth mentioning one way or another.