Subject: [Json] Security Considerations

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 12:17:02 -0700

Proposal:

    With any data format, it is important to encode correctly.  Care must
    be taken when constructing JSON texts by concatenation.  For example:

    account = 4627;
    comment = "\",\"account\":262";   // provided by attacker
    json_text = "(\"account\":" + account + ",\"comment\":\"" + comment 
+ "\"}";

    The result will be

    {"account":4627,"comment":"","account":262}

    which some parsers MAY see as being the same as

    {"comment":"","account":262}

    This confusion allows an attacker to modify the account property or
    any other property.

    It is much wiser to use JSON generators, which are available in many
    forms for most programming languages, to do the encoding, avoiding
    the confusion hazard.

    JSON is so similar to some programming languages that the native
    parsing ability of the language processors can be used to parse JSON
    texts.  This should be avoided because the native parser will accept
    code which is not JSON.

    For example, JavaScript's eval() function is able parse JSON text,
    but is can also parse programs.  If an attacker can inject code into
    the JSON text (as we saw above), then it can compromise the system.
    JSON parsers should always be used instead.

---

From: Stephan Beal <[email-masked]>
Date: Thu, 6 Jun 2013 22:00:56 +0200

>   {"comment":"","account":262}

That wording "may" not be valid/necessary, depending on how the debate on
the handling of multiple keys goes ;). If, for example, implementations are
required to fail on multiple keys then any other behaviour is non-compliant
and need not (necessarily) be specified by the spec, except perhaps as
"non-normative" info.

>    forms for most programming languages, to do the encoding, avoiding

Some generators (streaming ones) can still create dupes.

>    JSON parsers should always be used instead.

IMHO that all belongs well within the realm of Best Practice (perhaps a
Security Concerns sub-section).

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: "John Levine" <[email-masked]>
Date: 6 Jun 2013 23:02:39 -0000

I'd suggest also adding some warnings to parser writers along these
lines:

A string presented to a json parser may be a well formed json object,
or it may be anything else.  Hence a parser needs to avoid any
assumptions that its input is well-formed.  Even if an input string is
syntactically valid, strings may be longer than the maximum length of
an internal string format, numbers may not be representable in an
internal numeric format, arrays may be longer than the limits of an
internal array format, and an object may include more members than an
internal form can represent. Objects and arrays may be more deeply
nested than an internal form can represent.

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 6 Jun 2013 16:04:25 -0700

<no hat>

On Jun 6, 2013, at 12:17 PM, Douglas Crockford <[email-masked]> wrote:

>   json_text = "(\"account\":" + account + ",\"comment\":\"" + comment + "\"}";
> 
>   The result will be
> 

The example is language-specific and, due to the escaping, hard to read. Proposed replacement:

Programs that construct JSON texts by concatenating text from user input need to take care that the result is unsurprising. For example, when creating an object, a user might be able to inject a value that would override another value that is being put into the object. JSON's escaping rules for text make such attacks particularly difficult to prevent.

>   JSON parsers should always be used instead.

Much better than what we have now.

--Paul Hoffman

---

From: Douglas Crockford <[email-masked]>
Date: Thu, 06 Jun 2013 16:10:54 -0700

On 6/6/2013 4:04 PM, Paul Hoffman wrote:
> The example is language-specific and, due to the escaping, hard to read.
Which specific language would you say it is? Confusion attacks are often 
hard to read. That is why they work.

---

From: Peter Brooks <[email-masked]>
Date: Fri, 7 Jun 2013 08:22:21 +0200

I see two quite separate issues here.

1. The security of JSON itself as a data exchange mechanism
2. The ability for organisations to exercise good governance by having
their defined security policy reflected transparently in the access
controls of objects

The first issue applies only where JSON is exchanged in cleartext. If
JSON messages are exchanged through secure, encrypted channels, such
as SSL, then the chance of any successful interception and
modification of the message without detection is very small and more
the responsibility of the security software than the structure of JSON
itself. If JSON messages are exchanged in cleartext, that, itself, is
an indication that they are public and unimportant, so the risk, even
if they are intercepted and modified, is low.

The second issue matters even if the JSON messages are encrypted. If
the software fails to enforce corporate security policy, objects can
be revealed to unauthorised parties without detection being easy, or
even possible, as the logic is in the application, not the data, the
JSON object itself.

It is possible to secure JSON at the moment by adding fields that do
one of two things:

- Apply a corporate confidentiality level to an object
('Public','Private','Company Confidential','Internal Only','Board
Members Only', for example). The names of the fields are not defined,
though, so a scheme set up by one company would not be compatible with
one set up by another, which would cause problems if the two merged.
This inconsistency also makes it more difficult to validate the
security mechanism.

- Define the roles that are allowed to have, or are denied, access to
an object. Again, this is possible today, but there is no agreement as
to what syntax the access controls should follow or how the roles
should be defined.

If a system for security control was added to JSON in this way, it
would have to be backwards compatible by having any object without a
confidentiality level or access control being treated as PUBLIC.

The virtue of embedding this in the objects, not the application, is:

- applications can be tested and validated against access controls, in
detail, and proved to be complying with the JSON security.

- security staff, auditors and others can validate the security of
objects against policy by examining only the JSON schema and JSON data
with no requirement (once they have been tested and validated as
above) to examine the applications themselves.

Both of these would lead to JSON being recommended as an acceptable
data exchange method for organisations requiring good security
governance.

Obviously, securing the JSON objects would not prevent JSON files
being exchanged with the secure data exposed, but, again, the security
of files or databases is the responsibility of other security matters,
not the definition of JSON.

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 09:14:39 +0200

On 07.06.13 01:10, Douglas Crockford wrote:
> hard to read. That is why they work. ...

I propose to keep the example, but remove the need for quoting by 
switching to single quotes where needed:

"""
account = 4627;
comment = '","account":262';  // provided by attacker
json_text = '("account":' + account + ',"comment":"' + comment + '"}';
"""

this is valid (Java|ECMA)Script, right and also much more readable, 
isn't it?

All the best,
Stefan.

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 09:26:22 +0200

On 06.06.13 21:17, Douglas Crockford wrote:
>     ...

as JSON allows '{"\/":null}' this already forbids any native parsing 
ability of C like (or "based") languages including Python ;-)

I think these paragraphs are a very good change, but might ripen even 
more through discussions.

For me the main points of data interchange and in sequence are:

1. Do not parse any untrusted data without preconditioning

2. Do not excecute/eval untrusted data

and trust is sometimes hard to establish ...

As long as these two points stand out and are not burried inside sample 
language terms, I am happy with any re-phrasing.

Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 Jun 2013 08:58:48 -0700

On Jun 7, 2013, at 12:14 AM, Stefan Drees <[email-masked]> wrote:

> I propose to keep the example, but remove the need for quoting by switching to single quotes where needed:
> 
> """
> account = 4627;
> this is valid (Java|ECMA)Script, right and also much more readable, isn't it?

I can live with this, but I think doing it in English wording is better.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 Jun 2013 08:59:46 -0700

On Jun 7, 2013, at 12:26 AM, Stefan Drees <[email-masked]> wrote:

> As long as these two points stand out and are not burried inside sample language terms, I am happy with any re-phrasing.

That sounds like a good goal for the section.

--Paul Hoffman

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 18:36:51 +0200

> I can live with this, but I think doing it in English wording is better.

Proposal in English wording:

replace
OLD(++):
"""

account = 4627;

"""

with:
NEW:
"""
With any data format, it is important to encode correctly.  Care must
be taken when constructing JSON texts by concatenation in part from 
untrusted data, this data may inject a mix of structural characters and 
quotes, that changes the structure of the resulting JSON text.
For example: When trying to compose an object like
{"account": 4627, "comment": "foo"} from user input data for "foo", the 
attacking user might instead inject ","account":262,"comment":"hacked so 
that the resulting naively concatenated JSON text would become:
{"account": 4627, "comment": "","account":262,"comment":"hacked"}

"""

What do you think?

All the best,
Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 7 Jun 2013 14:32:01 -0700

On Jun 7, 2013, at 9:36 AM, Stefan Drees <[email-masked]> wrote:

> be taken when constructing JSON texts by concatenation in part from untrusted data, this data may inject a mix of structural characters and quotes, that changes the structure of the resulting JSON text.
> {"account": 4627, "comment": "foo"} from user input data for "foo", the attacking user might instead inject ","account":262,"comment":"hacked so that the resulting naively concatenated JSON text would become:

I'm liking that more, but we still have quoting problems. They can be solved by linebreaks.

For example: When trying to compose an object such as:
   {"account": 4627, "comment": "foo"}
from user input data for "foo", the attacker might instead inject the text:
   ","account":262,"comment":"hacked
so that the resulting naively concatenated JSON text would become:
   {"account": 4627, "comment": "","account":262,"comment":"hacked"}

--Paul Hoffman

---

From: John Cowan <[email-masked]>
Date: Fri, 7 Jun 2013 23:40:17 -0400

Peter Brooks scripsit:

> If JSON messages are exchanged in cleartext, that, itself, is

Open text is not necessarily unimportant text.  Consider these email
messages, which are sent en clair.

-- 
Newbies always ask:                             John Cowan
  "Elements or attributes?                      http://www.ccil.org/~cowan
Which will serve me best?"                      [email-masked]
  Those who know roar like lions;
  Wise hackers smile like tigers.                   --a tanka, or extended haiku

---

From: Peter brooks <[email-masked]>
Date: Sat, 8 Jun 2013 09:00:53 +0200

Sent from my iPad

> messages, which are sent en clair.
I agree. This, though, is bad practice. Any responsible organisation ought to encrypt its sensitive information.

---

From: Stefan Drees <[email-masked]>
Date: Sat, 08 Jun 2013 09:46:17 +0200

On 08.06.13 09:00, Peter Brooks wrote: ...
> On 8 Jun 2013, at 05:40, John Cowan ... wrote:
>> Peter Brooks scripsit:
>>
> I agree. This, though, is bad practice. Any responsible organisation
> ought to encrypt its sensitive information.

No, please do not label "any" cleartext exchange of broadly classified 
"sensitive information" as a generally "bad practice". For "secrets" ok, 
but "sensitive"? No, not in general.

Every communication has its setup phase, many communication channels 
over time change in the degree of what the impact of the transported 
data on the participants or others might be, even time may change the 
amount of "sensitivity" (like the words and deeds of a young person 
might interfere later with the same person later and in unintended ways).

Shaping the future JSON Format for Data Interchange is also possibly 
much more "sensitive" (as it impacts all conforming future JSON 
messages), than attacking the integrity, confidentiality, or flow of 
some JSON text instances based on that spec ;-)

All the best,
Stefan.

---

From: Peter brooks <[email-masked]>
Date: Sat, 8 Jun 2013 09:53:05 +0200

> 
> 
> 
> Shaping the future JSON Format for Data Interchange is also possibly much more "sensitive" (as it impacts all conforming future JSON messages), than attacking the integrity, confidentiality, or flow of some JSON text instances based on that spec ;-)
> 
I agree completely. That is why a proper security spec is important. 

You are quite right about current bad practice. It should be taken as a warning and reason to do better.

---

From: Stefan Drees <[email-masked]>
Date: Sat, 08 Jun 2013 10:16:30 +0200

>     {"account": 4627, "comment": "","account":262,"comment":"hacked"}

I like the indented line formating of the "inserted" or "produced" JSON 
sample portions.

So the proposal NEW is now:
"""
from user input data for "foo", the attacking user might instead inject 
the text:

"""

Please note: When discussing technical RFCs, I assume, that the final 
"to be submitted" paginated text with maximal line character count N 
determines a lot of the formatting. So I expect us to discuss the 
"message" as open group and later some persons remove the nits :-)

Stefan.

---

From: Eliot Lear <[email-masked]>
Date: Sat, 08 Jun 2013 15:03:59 +0200

>> 1. Do not parse any untrusted data without preconditioning

I think you include bounds testing in the above?

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

---

From: Stefan Drees <[email-masked]>
Date: Sat, 08 Jun 2013 15:32:55 +0200

On 2013-06-08 15:03, Eliot Lear wrote:
>> On Jun 7, 2013, at 12:26 AM, Stefan Drees ... wrote:
>>

Yes. As a goal I used the broad term "preconditioning" on purpose.
The devil is in the details. If one reads the latest draft text (as 
proposed by me) on parsers:

"""
4.  Parsers

    A JSON parser transforms JSON text into another representation,
    MUST accept all texts that conform to the JSON grammar and MUST be
    prepared to either accept duplicate names in objects or reject the
    complete JSON text containing these.
    A JSON parser MAY accept non-JSON forms or extensions.

    An implementation may set limits on any of the following: the size
    of texts that it accepts, the maximum depth of nesting, the range of
    numbers, and the length and character contents of strings.

"""

So many needed boundary conditions jump in view, where bounds testing is 
relevant to block attacks (from the above text):

1. the size of texts that it accepts,
2. the maximum depth of nesting,
3. the range of numbers, and
4. the length and character contents of strings

So IMO a JSON message receiver must be prepared to:

Ad 1.  reject after having received MAX_ACCEPT+1 bytes

Ad 2.a keep track of nesting counters CNT_NEST
Ad 2.b reject after CNT_NEST > MAX_NEST on first encounter

Ad 3.a first scan the CNT_BYTE_N presentation length of number cand
Ad 3.b reject if CNT_BYTE_N > SomeNumbersLiteralRepLen_MAX_BYTE
Ad 3.c all bytes of number collected match the range OK_RANGE_N
Ad 3.d reject if OK_RANGE_N is false

Ad 4.a scan the CNT_BYTE_STR of string cand
Ad 4.b reject if CNT_BYTE_STR > StringsLiteralRepLen_MAX_BYTE
Ad 4.c scan each byte and piddle, piddle all the rules of decoding
Ad 4.d reject on first encounter of invalid decoding request

I know with all the nitty gritty of say "\/" becomes "/" and other
length variations during the process from receipt to memory 
representation at parsing end, it is clear, that there must be a size 
limit up front and not "after encoding", which helps the parser to never 
have a "too full stomach" so to say.

That would be (as I said) my personal expectation on handling JSON text 
coming in from the wild.

The important JSON promise of "bounded" data interchange is thus in my 
view kept by applying the parsers maximum size functions (dealing with 
the limits of 1,2,3 and 4 above) on the bounds offered inside the 
generators JSON text the parser tries to scan and tokenize.

>> That sounds like a good goal for the section. ...

I think that the second aspect has become more and more accepted reality 
and is not as important as it was, when JSON was young, and the JS stood 
for JavaScript ;-)

All the best,
Stefan.

---

From: "John Levine" <[email-masked]>
Date: 9 Jun 2013 04:25:18 -0000

>    numbers, and the length and character contents of strings.

Probably should say something about the precision of numbers.  With a
limited precision internal representation, two numbers that are
different in the JSON could end up equal internally.

---

From: Stefan Drees <[email-masked]>
Date: Sun, 09 Jun 2013 10:49:27 +0200

> different in the JSON could end up equal internally.

yes, thus I propose as NEW:

"""
4.  Parsers

    An implementation may set limits on any of the following: the size
    of texts that it accepts, the maximum depth of nesting, the range
    and precision of numbers, and the length and character contents of
    strings.

"""

I think simply extending the caveat for numbers to range and precision
is sufficient, as integral numbers are to be considered having precision 
0 and it all starts with a "may" ;-)

Stefan.

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 9 Jun 2013 13:38:02 -0700

Starting this as a new thread.

---

From: Stefan Drees <[email-masked]>
Date: Mon, 10 Jun 2013 08:18:07 +0200

On 2013-06-09 22:38 CEST, Paul Hoffman wrote:
> On Jun 9, 2013, at 1:49 AM, Stefan Drees ... wrote:
>

As the discussion continued, I rewrote the proposal in [1] (cf. for 
notes and motivation there) suggesting the harshest possible "soft" 
guard against the "parsing and storage" scenario described by "not 
yielding the last occurence of multiple names, when only yielding one", 
"discussed" the "REALLY SHOULD always" of the second sentence of [1] in 
[2,3] with Nico and now hereby offer the freshest proposal for the 
Parsers section to start discussing in this new thread created by Paul:

NEW:
"""
4.  Parsers

    A JSON parser transforms JSON text into another representation,
    MUST accept all texts that conform to the JSON grammar and MUST be
    prepared to either accept duplicate names in objects or reject the
    complete JSON text containing these.
    If the JSON parser transforms name/value pairs of JSON objects
    into maps inside the target representation by using the received
    names as keys, then it MUST yield the values of the
    last encountered occurences of the name/value pairs as to satisfy
    the principle of least surprise.
    A JSON parser MAY accept non-JSON forms or extensions.

"""

So finally I throw a MUST into the ring! Let's try to describe currrent 
practice and - armed with precise details of the parsing and storage 
scenario we refer to - set an upper bound for ambiguity that started 
when JSON was born without explicitely stating the consequence of 
interpreting names (in objects) as keys. What do you think?

For a matching proposal for the Generators section please cf. [1].

References:

[1]: http://www.ietf.org/mail-archive/web/json/current/msg00689.html
[2]: http://www.ietf.org/mail-archive/web/json/current/msg00691.html
[3]: http://www.ietf.org/mail-archive/web/json/current/msg00692.html

All the best,
Stefan

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 10 Jun 2013 12:27:20 +0200

On Jun 9, 2013, at 22:38, Paul Hoffman <[email-masked]> wrote:

>> duplicate names in objects

That doesn't mean anything, until it has been defined.

Is "foofoo" a duplicate name?

Or is it just not allowed to write 
{ "foo" "foo" : "bar" }
?

Or is it that all maps (objects) must have unique names, so 
[{"foo": "bar"}, {"foo": "baz"}]
is an instance of a duplicate name?

Or maybe this is about the names within an object, so
{"foo": {"foo": "bar"}}
is an instance of a duplicate name?

->
Of the name/value pairs that comprise a single object, no two [shall/should/whatever] have the same name.
For the purposes of this comparison, two names are the same if they are the same sequence of characters after processing the strings (2.5).

[RFC 4627 doesn't quite define what that is, either, but one can complete section 2.5 with common knowledge such as that the quotes aren't meant to be part of the decoded string.]

Grüße, Carsten

---

From: Stefan Drees <[email-masked]>
Date: Mon, 10 Jun 2013 13:56:09 +0200

On 2013-06-10 12:27 CEST, Carsten Bormann wrote:
> On Jun 9, 2013, at 22:38, Paul Hoffman ... wrote:
>

I think here the contxt is confined to parsers and is a summarizing 
description with focus on the parsers. I understand the job of the 
section 2 to introduce the vocabulary. So the name is the first part of 
the ordered pair (name, value).

> Is "foofoo" a duplicate name?

not in itself, but maybe if some other "foofoo" in the same object and 
nesting level comes along or preceeded it.

> Or is it just not allowed to write

here the parser is already out, as this is invalid JSON.

> is an instance of a duplicate name?

we should talk about names of a single object and nesting level.
So no.

> is an instance of a duplicate name?

No. ditto.

> ->
> Of the name/value pairs that comprise a single object, no two
> [shall/should/whatever] have the same name.
> For the purposes of this comparison, two names are the same if they
> are the same sequence of characters after processing the strings (2.5).

As long as we do not "invent" something, but are all sure, that the 
"common" knowledge where it fills a real gap has also been "shared by 
all known implementers" knowledge :-) ...

> [RFC 4627 doesn't quite define what that is, either, but one can
> complete section 2.5 with common knowledge such as that the quotes
> aren't meant to be part of the decoded string.] ...

How do the existing validators and parsers decide that {"a" "a":"b"}
is invalid maybe the ABNF rules :-?)
"""
          object = begin-object [ member *( value-separator member ) ]
                 end-object
          member = string name-separator value
[...]
          string = quotation-mark *char quotation-mark
          char = unescaped /
                 escape (
                     %x22 /          ; "    quotation mark  U+0022
                     %x5C /          ; \    reverse solidus U+005C
                     %x2F /          ; /    solidus         U+002F
                     %x62 /          ; b    backspace       U+0008
                     %x66 /          ; f    form feed       U+000C
                     %x6E /          ; n    line feed       U+000A
                     %x72 /          ; r    carriage return U+000D
                     %x74 /          ; t    tab             U+0009
                     %x75 4HEXDIG )  ; uXXXX                U+XXXX

          escape = %x5C              ; \

          quotation-mark = %x22      ; "

          unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

"""
where
     "a" "a"
seems not to match the rule for a string in rule member.

All the best,

Stefan.