Subject: [Json] Go JSON parser ignores the case of member names

From: "Manger, James" <[email-masked]>
Date: Wed, 9 Mar 2016 12:05:13 +1100

I noticed that the default JSON parsing in the Go language basically ignores the case of member names. Do JSON experts see this case-insensitive parsing as a useful convenience for programmers (being lenient with what you receive), or as a bad insecure design choice?

Parsing {"typ":"JWS","alg":"HS256","ALG":"none"} with the following code, for instance, prints "none" (not "HS256") - contrary to what the programmer wanted.

type Header struct {
                Alg string `json:"alg"`
                Typ string `json:"typ"`
}

func main() {
                b := []byte(`{"typ":"JWS","alg":"HS256","ALG":"none"}`)
                var h Header
                err := json.Unmarshal(b, &h)
                fmt.Println(h)
}

This looks quite diabolical for security as it is trivial to create valid JSON values that will be interpreted differently by different implementations. I would expect most implementations (that are expecting an "alg" member) to see its "HS256" value and simply ignore the extra "ALG" member. Not Go, however. Note: An alternative implementation in Go could parse the JSON as a generic map, then looked for the "alg" member, which would give "HS256".

The relevant part of the Go documentation for the encoding/json<https://golang.org/pkg/encoding/json/#Unmarshal> package is:
  "To unmarshal JSON into a struct, Unmarshal matches incoming object keys to the keys used by Marshal (either the struct field name or its tag), preferring an exact match but also accepting a case-insensitive match."

--
James Manger

---

From: John Cowan <[email-masked]>
Date: Tue, 8 Mar 2016 20:20:28 -0500

Manger, James scripsit:

> I noticed that the default JSON parsing in the Go language
> basically ignores the case of member names. Do JSON experts see this
> case-insensitive parsing as a useful convenience for programmers (being
> lenient with what you receive), or as a bad insecure design choice?

I see it as Very Bad Indeed.  It violates the RFC and, as you say, causes
a JSON text to be interpreted one way by one recipient and another way
by another.

> This looks quite diabolical for security as it is trivial to create
> valid JSON values that will be interpreted differently by different
> implementations. I would expect most implementations (that are expecting
> an "alg" member) to see its "HS256" value and simply ignore the extra
> "ALG" member. 

The Right Thing is to _validate your JSON_.  Make sure what you expect
is there *before* acting on it, and if not *don't act on it*.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
My confusion is rapidly waxing
For XML Schema's too taxing:
I'd use DTDs / If they had local trees --
I think I best switch to RELAX NG.

---

From: Rob Sayre <[email-masked]>
Date: Tue, 8 Mar 2016 19:36:10 -0800

Well, there are ways to avoid it too (see below). I see it more as an
object-mapper convenience than a JSON conformance problem. I imagine object
mapping libraries in other languages have this feature too.

- Rob

Program that prints all values from example:

package main

import (
"encoding/json"
"log"
)

func main() {
b := []byte(`{"typ":"JWS","alg":"HS256","ALG":"none"}`)
m := make(map[string]string)
err := json.Unmarshal(b, &m)
if err != nil {
log.Fatal("JSON parsing failed")
}
for k, v := range m {
println("k:", k, "v:", v)
}
}

---

k: typ v: JWS
k: alg v: HS256
k: ALG v: none

On Tue, Mar 8, 2016 at 5:20 PM, John Cowan <[email-masked]> wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Mar 2016 16:38:31 -0500

> mapping libraries in other languages have this feature too.

The problem arises in cases like this: Suppose the spec says that key
"alg" specifies the name of the algorithm, and defaults to "none" if
there is no key.  Then if the upstream provider sends {"typ" : "JWS",
"ALG": "HS256"}, then a receiver written in Go will see the algorithm
as HS256, whereas any other receiver will see the algorithm as "none".
That's almost certainly not the desired result.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
To say that Bilbo's breath was taken away is no description at all.  There are
no words left to express his staggerment, since Men changed the language that
they learned of elves in the days when all the world was wonderful. --The Hobbit

---

From: Anders Rundgren <[email-masked]>
Date: Wed, 9 Mar 2016 23:12:56 +0100

On 2016-03-09 22:38, John Cowan wrote:
> as HS256,

Which indeed is bad.

> whereas any other receiver will see the algorithm as "none".

Using a RealJSONParser(tm) other receivers will rather choke on the message
and complain about an unexpected "ALG" property.

I'm just in the process adding such a facility to node.js:
https://github.com/cyberphone/node-webpki.org/blob/master/lib/jsonutil.js

Anders

---

From: Rob Sayre <[email-masked]>
Date: Wed, 9 Mar 2016 18:01:02 -0800

> That's almost certainly not the desired result.

I understand the situation. I guess I don't agree there's a conformance
problem here, since there are plenty of other APIs in the standard package
that don't do this transformation, and even the API in question doesn't do
this transformation for all arguments. They're right below the section
James quoted.

BTW, here's how to get a similar effect using the widely-used Java library
Jackson[0]:

ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES,
true);

- Rob

[0] http://wiki.fasterxml.com/JacksonHome

---

From: John Cowan <[email-masked]>
Date: Wed, 9 Mar 2016 21:15:02 -0500

> this transformation for all arguments. They're right below the section

Ah.  I didn't know that.

> objectMapper.configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES,

Sure, but this is a feature you have to enable specifically, not the
default.  But from what you are saying, the Go case mapping is not the
default either.

It's perfectly reasonable, given how much ill-formed or incorrect JSON
there is, to have relaxed modes.  It's just that the main mode should
be strict.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Verbogeny is one of the pleasurettes of a creatific thinkerizer.
        --Peter da Silva

---

From: "Manger, James" <[email-masked]>
Date: Thu, 10 Mar 2016 15:55:47 +1100

>> I understand the situation. I guess I don't agree there's a 
>> conformance problem here, since there are plenty of other APIs in the 
>> standard package that don't do this transformation, and even the API 
>> in question doesn't do this transformation for all arguments. They're 
>> right below the section James quoted.

> Ah.  I didn't know that.

I am fairly new to Go so there is heaps I don't know. However, I don't think the case-insensitive transformation is quite as optional as Rob suggests.

"encoding/json" looks like THE standard JSON package in Go.
It can parse (Unmarshall) a JSON object into a Go map or struct.
Parsing to a map (useful for handling unknown or arbitrary JSON) preserves the case of the object members.
Parsing to a struct (ie object-mapping) ignores case - always.

I cannot see any option NOT to ignore case when parsing to a struct. There is no equivalent of MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES.

Object-mapping is a great developer convenience. However, it looks like the Go implementation is an "attractive nuisance" < https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine>. Parsing to a struct is front and centre in the doco, yet comes with this risky case-ignoring transformation that cannot be disabled.

>> BTW, here's how to get a similar effect using the widely-used Java library Jackson[0]:
>> 
>> objectMapper.configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES, true);

> Sure, but this is a feature you have to enable specifically, not the default.  But from what you are saying, the Go case mapping is not the default either.

> It's perfectly reasonable, given how much ill-formed or incorrect JSON there is, to have relaxed modes.  It's just that the main mode should be strict.

Go supports tags on struct fields (eg Algorithm string `json:"alg,omitempty"`). Supporting, say, an "ignorecase" option in the tag might have been a reasonable design.

P.S. Another interesting feature of the standard JSON package in Go is that it ALWAYS escapes "<" as "\u003c" - as a safety feature, in case the JSON is used within HTML. This safety decision seems such a contrast to the decision to ALWAYS ignore case when parsing to a struct.

--
James Manger

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 10 Mar 2016 07:54:12 +0100

On 2016-03-10 05:55, Manger, James wrote:
<snip>
> P.S. Another interesting feature of the standard JSON package in Go is that it
 > ALWAYS escapes "<" as "\u003c" - as a safety feature, in case the JSON is used
 > within HTML. This safety decision seems such a contrast to the decision to ALWAYS ignore case when parsing to a struct.

Thanx for letting me know!

The Go team obviously has a lot to do if they want to be compatible with ES6
and be able to create "crypto-safe" JSON objects like required by JCS:
https://github.com/golang/go/issues/14135#issuecomment-177265555

Microsoft are closing in on this AFAICT:
https://github.com/dotnet/corefx/pull/6665#issuecomment-192380817

Anders
https://cyberphone.github.io/openkeystore/resources/docs/jsonsignatures.html

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 10 Mar 2016 11:13:40 +0100

On 2016-03-10 05:55, Manger, James wrote:

> P.S. Another interesting feature of the standard JSON package in Go is that it ALWAYS escapes
> "<" as "\u003c" - as a safety feature, in case the JSON is used within HTML.

Whow James, you are a true genius for finding difficult spots!

My JSON tools have be upgraded to support JSON-serializable JavaScript objects
intended for usage in Web pages.  That's great but (of course...) I had overlooked
the case you mentioned:

<html...>

<script>

var jsonData = {
   trouble: "</script>"
};

FAIL!

That is, you really have to do as Go which fortunately have no practical implications
since an ES6 compatible serializer will anyway produce the right (normalized) result
during signature verification.  Sometimes you need a little bit of luck as well...

Anders

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 10 Mar 2016 13:57:39 +0100

On 2016-03-10 11:13, Anders Rundgren wrote:
https://github.com/golang/go/issues/14749