Subject: [OAUTH-WG] Why JSON (Was: [oauth] Re: [twitter-dev] Hard lesson learned)

From: Dick Hardt <[email-masked]>
Date: Sun, 23 May 2010 15:32:05 -0700

Frustration devs have with URL encoding. This is the core motivation to using JSON as the AS response format.

-- Dick

Begin forwarded message:

> From: Andrew Badera <[email-masked]>
> Date: May 23, 2010 2:57:44 PM PDT
> To: [email-masked]
> Cc: [email-masked]
> Subject: [oauth] Re: [twitter-dev] Hard lesson learned
> Reply-To: [email-masked]
> 
> Miguel,
> 
> This 'lesson' has been 'learned' and re-learned many times over, here on the Twitter dev list and on the oauth list. One would hope that at some point this issue would rise to enough prominence to get people in charge of implementation, and sig participants in general, to do something about it. The common developer these days is not a super savvy geek, and even the super savvy geeks among us waste time on this issue, again and again.
> 
> ∞ Andy Badera
> ∞ +1 518-641-1280 Google Voice
> ∞ This email is: [ ] bloggable [x] ask first [ ] private
> ∞ Google me: http://www.google.com/search?q=andrew%20badera
> 
> 
> On Sun, May 23, 2010 at 5:52 PM, Miguel de Icaza <[email-masked]> wrote:
> Hello guys,
> 
>    Perhaps the most frustrating piece in dealing with the OAuth
> configuration is that the twitter OAuth page talks casually about
> "urlEncode".  You need to "urlEncode this" and "urlEncode that".  What
> the page does not say is that "urlEncode" is not a standard
> urlEncoding system that web developers are used to.  The urlEncode
> required by OAuth signatures is actually "percent encode" and it is
> *required* that you use percent encoding for anything but a small
> subset of characters.
> 
>    The only characters that do not require percent encoding are:
> 
> unreserved = a through z, A through Z, 0 through 9 and  '-', '.', '_',
> '~'
> 
> Miguel
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "OAuth" group.
> To post to this group, send email to [email-masked].
> To unsubscribe from this group, send email to [email-masked].
> For more options, visit this group at http://groups.google.com/group/oauth?hl=en.

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Mon, 24 May 2010 14:13:11 -0700

This is a bad example. This deals with encoding for the purpose if the signature base string. Not the same as decoding responses which I doubt is a problem for Twitter as I think their token values are URL safe.

EHL

On 5/23/10 3:32 PM, "Dick Hardt" <[email-masked]> wrote:

-- Dick

Begin forwarded message:

Miguel,

? Andy Badera
? +1 518-641-1280 Google Voice
? This email is: [ ] bloggable [x] ask first [ ] private
? Google me: http://www.google.com/search?q=andrew%20badera

Miguel

---

From: Marius Scurtescu <[email-masked]>
Date: Mon, 24 May 2010 16:55:13 -0700

And to add to this, this example shows that encoding is hard, JSON
only solves decoding (in most cases, but not all).

For all direct requests clients still need to encode and without a
library still need to figure what chars must be encoded.

Introducing JSON because dealing with form-encoded is hard does not
make much sense. As discussions last week showed (and earlier on this
list), with JSON there is also the perception that it is easy to do by
hand and that no escaping is needed. IMO that will lead to way more
problems.

I guess that there are other reasons to use JSON. The best argument I
heard is that JSON may be used in some token formats and in discovery
documents. That could make sense, but even then, I don't think there
is any harm in using form-encoded for the basic protocol.

Marius

On Mon, May 24, 2010 at 2:13 PM, Eran Hammer-Lahav <[email-masked]> wrote:
> This is a bad example. This deals with encoding for the purpose if the
> signature base string. Not the same as decoding responses which I doubt is a
> problem for Twitter as I think their token values are URL safe.
>
> EHL
>
>
> This 'lesson' has been 'learned' and re-learned many times over, here on the
> Twitter dev list and on the oauth list. One would hope that at some point
> this issue would rise to enough prominence to get people in charge of
> implementation, and sig participants in general, to do something about it.
> The common developer these days is not a super savvy geek, and even the
> super savvy geeks among us waste time on this issue, again and again.
>
> ∞ Andy Badera
> _______________________________________________
> OAuth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/oauth
>
>

---

From: Dick Hardt <[email-masked]>
Date: Mon, 24 May 2010 17:52:53 -0700

> only solves decoding (in most cases, but not all).

JSON solves encoding and decoding with the same library.

> library still need to figure what chars must be encoded.

You argued this point numerous times at the in-person meeting. Clients are used to encoding and creating requests to servers. That is what they do. There are many simple ways of doing this.

Clients in general do NOT decode forms data, that is done by servers, and is built into server frameworks.

> hand and that no escaping is needed. IMO that will lead to way more

Libraries exit for encoding and decoding JSON. They are really easy to use. There is asymmetry in the forms data. Servers typically decode, and clients typically encode.

-- Dick

---

From: Leah Culver <[email-masked]>
Date: Mon, 24 May 2010 21:03:58 -0700

To me this smells a lot like XML a few years ago... how XML is baked into
XMPP and people thought that HTML should be converted to XML... seemed like
a good idea at the time :D

So yeah, I'm against supporting ONLY JSON. I'd be fine if the spec allowed
for form-encoded, XML or whatever else might become the hot new thing. I'm
really for leaving it up to the provider. I don't think it's up to OAuth to
force people to use what we think is the coolest encoding format.

Leah

On Mon, May 24, 2010 at 5:52 PM, Dick Hardt <[email-masked]> wrote:

> You argued this point numerous times at the in-person meeting. Clients are
> used to encoding and creating requests to servers. That is what they do.
> There are many simple ways of doing this.
>
> Libraries exit for encoding and decoding JSON. They are really easy to use.
> There is asymmetry in the forms data. Servers typically decode, and clients
> typically encode.
>
> -- Dick

---

From: Pid <[email-masked]>
Date: Tue, 25 May 2010 09:40:43 +0100

> Clients in general do NOT decode forms data, that is done by servers, and is built into server frameworks.

If one was feeling mischievous, one could argue that an implementer only
needs to find or create a library for the client side, rather than both
- like JSON.

> Libraries exit for encoding and decoding JSON. They are really easy to use. There is asymmetry in the forms data. Servers typically decode, and clients typically encode.

I really don't think this is a reasonable argument in favour of JSON.
It's somewhat confusing to argue in favour of using an external library
on one hand, but not the other.

The best summary of the arguments in favour of JSON was by Joseph Smarr
in response to my previous inquiry, which hasn't been superseded, I think.

However, IMO JSON is yet to be demonstrated as the better solution.

p

---

From: Kris Selden <[email-masked]>
Date: Tue, 25 May 2010 13:51:08 -0700

Once again, I want to plea the case for keeping the response simple key/value string pairs so it is trivial to serialize to multiple formats. The OAuth endpoint should be able to match the format(s) of the API it protects.

Given this simple response payload, the spec can generically describe how to serialize the string key/value pairs to application/x-www-form-urlencoded, application/json and application/xml with other formats added through extensions.

Values that are lists can simply be a delimited string, with the delimiter either a character that cannot be in the value (like space delimited urls) or use a simple escaping mechanism described by the spec. Integers can be spec'd as such and not rely on format spec. I know, this means in JSON {"expires_in":"3600"} which gives people a rash but helps make it trivial to decouple format.

String encoding should be specified by the charset parameter of the Content-Type header of the response. The client could indicate supported charsets through Accept-Charset in the request. Likely this should match how it is handled in the protected APIs. Anyway, let me digress a little, which responses from an OAuth endpoint are even likely to run into any problems with encoding -- what needs more than ASCII? Even if a bearer token is in JSON you could still use \u escapes on a UTF8 encoded JSON to reduce it to ASCII.

Some of the arguments for JSON as the only format remind me of an infomercial where they make a basic kitchen implement look harder to use than it is, to exaggerate the need of their handy dandy slicer dicer tool.

No, there aren't a lot of libraries that do form decoding on the client side (if it is a language that is only typically used on the client side, is this anything but JavaScript or ActionScript?) but I can't think of any that do not have decoding for percent encoding (which isn't hard anyway) and the rest is trivial to parse.

The headaches in OAuth 1.0 came from the signature base string's use of percent encoding which is a separate issue from using application/x-www-form-urlencoded as a response type.

Making the OAuth endpoint any one format (unless that response type is truly trivial to parse without 3rd party libraries) renders an APIs support of multiple formats (which seems quite popular) pointless.

---

From: Marius Scurtescu <[email-masked]>
Date: Tue, 25 May 2010 18:19:55 -0700

> in response to my previous inquiry, which hasn't been superseded, I think.

Yes, Joseph made a very good argument for JSON.

My only objection is that form-encoded is still used in several
places, so the problems are still not solved, and now clients have to
deal with 2 encodings. I know that there is disagreement on this, but
I have not seen a good proof that this is not true.

Marius

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 26 May 2010 17:19:54 +1000

Kris,

> The OAuth endpoint should be able to match the format(s) of the API it 
> protects.

I had a quick look at the APIs for 24 services implementing OAuth (v1 or 
drafts of v2) as listed on the OAuth wiki.

Of these 24 APIs, the response formats supported were:
* 21 XML
* 19 JSON
* 16 XML and JSON
*  5 XML only
*  3 JSON only

Other formats with some support: Atom, RSS, PHP, JSONP, CSV, vCards, 
FastInfoset, XML-RPC.

None use form-encoding for responses.

> Once again, I want to plea the case for keeping the response
> simple key/value string pairs so it is trivial to serialize to multiple 
> formats.

Most APIs from this sample offer multiple formats.
None restricted themselves to simple key/value string pairs.
None attempt to serialize to a format with as little structure as 
form-encoding.

Where OAuth2 differs from most APIs is that it offers a response over 2 
different "channels": an HTTP response body (as APIs do); and in a HTTP 
redirect (which most APIs don't do).
If we really want to match the formats of the API being protected, then the 
HTTP redirect channel should tunnel exactly the same response that could be 
obtained in an API-like response body. Putting base64url-encoded XML or JSON 
in the redirect URI is one way to do such tunnelling. Its not the simplest 
solution for the simplest use case, but it is clear and future-proof.

--
James Manger

---

From: Kris Selden <[email-masked]>
Date: Wed, 26 May 2010 03:06:46 -0700

I like your response.

I think application/json and application/xml would be fine without application/x-www-form-urlencoded.  I just thought it made a good lowest common denominator for serialization which I think aids in making it simple to support multiple formats.

---

From: Dan Brickley <[email-masked]>
Date: Wed, 26 May 2010 12:27:15 +0200

> force people to use what we think is the coolest encoding format.

Well said! "At least JSON" would be fine, but leave the door open for
other formats...

Dan