Subject: [Tools-discuss] RESTful API style guide

From: Michael Richardson <[email-masked]>
Date: Mon, 17 Jun 2013 21:57:52 -0400

As part of the Agenda Tool work, a number of JSON API calls have been
created.  I have started writing some document for them, at least to have
a list of what is there.  There are a number of inconsistencies in the API,
some of which I know how I want to fix, and some which I do not...

I'm writing it in markdown, although I found that neither the perl nor python
markdown converter seems to deal properly with code samples indented by 4
spaces.  Any suggestions on a better tool chain?

Once I have written up what there is, I will be very interested in what parts
are just broken.  To whet your appetite, the Agenda Tool loads many group
descriptions like:

[17/Jun/2013 18:04:50] "GET /group/splices.json HTTP/1.1" 200 531
[17/Jun/2013 18:04:50] "GET /group/raiarea.json HTTP/1.1" 200 378
[17/Jun/2013 18:04:50] "GET /group/6lowpan.json HTTP/1.1" 200 554
[17/Jun/2013 18:04:50] "GET /group/autoconf.json HTTP/1.1" 200 620
[17/Jun/2013 18:04:50] "GET /group/tsvarea.json HTTP/1.1" 200 346
[17/Jun/2013 18:04:50] "GET /group/hip.json HTTP/1.1" 200 559
[17/Jun/2013 18:04:50] "GET /group/mip4.json HTTP/1.1" 200 530

This is just *TOO SLOW*.  Too many small requests.  (I assume the browser has
a 1.1-persistent session open, but I haven't checked actually).
I'd like to replace it with:
    GET /groups.json
returning an array of groups.  Does this make sense, architecturally?

Any suggestions on things to include, not include, in the API template?

Here is the rough draft of the first entry:

/meeting/(?P\d+).json$'

   Methods supported: GET, PUT, POST.

  GET method.

   Returns JSON description of the meeting.
    {
        "agenda_href": "http://thesiteurl/meeting/83/agendas/mtg:83.json",
        "break_area": "Hall Maillot A",
        "city": "Paris",
        "country": "FR",
        "date": "2012-03-25",
        "href": "http://thesiteurl/meeting/83.json",
        "name": "83",
        "reg_area": "Hall Maillot A",
        "submission_correction_date": "2012-05-16",
        "submission_cut_off_date": "2012-04-27",
        "submission_start_date": "2011-12-26",
        "time_zone": "Europe/Paris",
        "venue_name": "",
        "venus_addr": ""
    }

    href

   Canonical URL of this object.

    agenda_href

   URL for the official agenda.

    break_area

   What room are the breaks in.

  PUT method.

   Accepts JSON or "application/x-www-form-urlencoded" dictionary. Accepts
   one argument which is the name of the agenda that should become
   official. The special name "None" or an empty string will set the
   agenda empty.

   This call must be authenticated by a user with secretariat role, and
   the agenda must be already marked public, or an HTTP 406 will be
   returned.

   The result of this call, if successful, is the updated JSON of the
   meeting.

  POST method.

   Identical at present to the PUT method.

--
Michael Richardson <[email-masked]>, Sandelman Software Works

---

From: Martin Thomson <[email-masked]>
Date: Mon, 17 Jun 2013 21:06:11 -0700

> returning an array of groups.  Does this make sense, architecturally?

Yes.  That index can include any information you deem to be useful in
that context, which could be all fields.

I don't think that you have the PUT thing right, but I'd need more
context on what you are trying to achieve with that.  And probably a
more complete description of what happens (i.e., where the PUT is
targetted, not the URI, but the type of thing that you are PUTting
at.)  What you described sounds like a POST, not a PUT.

---

From: Ole Laursen <[email-masked]>
Date: Tue, 18 Jun 2013 15:08:15 +0200

> returning an array of groups.  Does this make sense, architecturally?

IMHO, you've got this question backwards - it doesn't make sense
architecturally to access each individually. :)

>    The result of this call, if successful, is the updated JSON of the

Agree with Martin that this is not a PUT, it's a POST. Also, if the
purpose is so specific, you should put it under a separate URL.
Furthermore, it sounds like this is an internal function to be used by
a Secretariat tool that presumably already has access to the database
so why are you even exposing it?

Ole

---

From: Tom Pusateri <[email-masked]>
Date: Tue, 18 Jun 2013 08:22:10 -0700

> architecturally to access each individually. :)

Maybe not for a web page that shows the whole agenda but there are mobile cases where it might make sense. It also might make sense to query all groups by date. But groups are not the only items on the agenda. There are lots of special sessions that are not groups. Querying these by the uuid in the iCal file would be helpful. I've been looking for a way to get the descriptions of the special sessions but apparently, they are only available from the database.

Thanks,
Tom

---

From: Michael Richardson <[email-masked]>
Date: Tue, 18 Jun 2013 15:45:18 -0400

Thanks for the comments on the substance.
I wanted to initially just get some comments on the *presentation*

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Ole Laursen <[email-masked]>
Date: Wed, 19 Jun 2013 12:24:03 +0200

2013/6/18 Tom Pusateri <[email-masked]>:
> Maybe not for a web page that shows the whole agenda but there are
> mobile cases where it might make sense. It also might make sense
> to query all groups by date.

I think it's better to provide and consume a batch API that allows one
to do a simple search, e.g. by encoding the search with a flexible GET
parameter scheme rather than in the (unflexible) path of the URL. This
is where many "REST" frameworks go wrong IMHO - how can you do cool
things if you start off in a straitjacket? Of course, YMMV.

Ole

---

From: Ole Laursen <[email-masked]>
Date: Wed, 19 Jun 2013 12:40:11 +0200

> I wanted to initially just get some comments on the *presentation*

It seems fine to me, but for the GET part maybe it's easier to just
refer people to a test URL where they can check the JSON, otherwise
you risk that it gets out of sync. Ideally the attributes would be
self-evident from their names in the JSON (or changed so they are),
but otherwise listing those that need explanation seems fine to me (as
long as the explanation adds new non-self-evident information :).

Are you going to add this to the trac wiki?

Ole

---

From: Michael Richardson <[email-masked]>
Date: Wed, 19 Jun 2013 10:29:23 -0400

Ole Laursen <[email-masked]> wrote:
    ole> I think it's better to provide and consume a batch API that allows
    ole> one to do a simple search, e.g. by encoding the search with a
    ole> flexible GET parameter scheme rather than in the (unflexible) path
    ole> of the URL. This is where many "REST" frameworks go wrong IMHO - how
    ole> can you do cool things if you start off in a straitjacket? Of
    ole> course, YMMV.

The list and contents of groups do not change often, they ought to be very
cachable.   If django supported page caching the way rails does, it could
be page cached, and therefore served by the apache in front.

A search parameter in a GET requires active work to process and reply, and is
uncachable.  Basically you are arguing for:
        http://foo/groups?name
vs
        http://foo/groups/name

---

From: Michael Richardson <[email-masked]>
Date: Wed, 19 Jun 2013 10:31:07 -0400

Ole Laursen <[email-masked]> wrote:
    >> Thanks for the comments on the substance.  I wanted to initially just
    >> get some comments on the *presentation*

    olau> It seems fine to me, but for the GET part maybe it's easier to just
    olau> refer people to a test URL where they can check the JSON, otherwise
    olau> you risk that it gets out of sync. Ideally the attributes would be
    olau> self-evident from their names in the JSON (or changed so they are),
    olau> but otherwise listing those that need explanation seems fine to me
    olau> (as long as the explanation adds new non-self-evident information
    olau> :).

I agree with adding a test URL, when the test URL is stable and deployed.
I don't propose to document anything that isn't self-evident.

    olau> Are you going to add this to the trac wiki?

Yes, that would make sense. Hmm. so really I should keep it in trac wiki
format.

---

From: Ole Laursen <[email-masked]>
Date: Wed, 19 Jun 2013 17:56:09 +0200

Hi Michael!

Sorry if I sounded rude, no offense intended, it's just that I
interpreted your first email as if you'd like feedback on the
architecture you've chosen, and that's now turned into a general
discussion about distributed APIs which I find interesting. To me, it
matters that the API doesn't encourage you to do to a lot of
roundtrips - that I think is one of the main lessons of CORBA (and X11
and ...).

Anyway, your comments:

> be page cached, and therefore served by the apache in front.

AFAIK, Django supports all the forms of caching you'd want. The Django
apps we run at IOLA are behind nginx which has a very simple reverse
proxy cache that you can control simply by the standard caching
headers. I'm pretty sure Apache has the same.

> uncachable.

I disagree with that. You don't really need to do more work compared
to what you're doing anyway (i.e. parse the acronym and do a database
query) and all GETs are cacheable by design, at least if you set the
right header.

> Basically you are arguing for:
>         http://foo/groups?name
> vs
>         http://foo/groups/name

I'm arguing for

http://foo/groups/?acronym=name1,name2,...

instead of

http://foo/groups/name1/
http://foo/groups/name2/
...

The former interface supports everything the latter does, it's more
flexible (just slap another GET parameter on it to implement search),
easier to use if you need more than one, and faster except in the case
where you only need 1. Hence, I don't see much point in doing the
latter interface - you increase your API surface with little value in
return.

Now, don't get me wrong, YMMV, URL paths work really well for HTML and
the web in general. But web documents are big, contain lots of
dependencies, and you usually only want one at the time so it's an
entirely different game.

I'm sorry if my thoughts weren't entirely clear, with this
clarification I'll shut up and get back to work,

Ole

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Jun 2013 18:47:41 +0200

> spaces.  Any suggestions on a better tool chain?

I generally use a tool called kramdown for my markdown needs.
(Including https://github.com/cabo/kramdown-rfc2629 -- but I gather this is not becoming an Internet-Draft?)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Jun 2013 11:49:47 -0500

> roundtrips - that I think is one of the main lessons of CORBA (and X11

And NFS...

This is true, APIs should aim to optimize for round trips.

But at higher layers APIs should be asynchronous/batching, so that any
round trips don't kill responsiveness.  This is *much* easier to
accomplish nowadays in the world of HTTP APIs because there has been
less stability at much higher layers (e.g., no opendir() + readdir() +
loop to stat(), getpwuid(), getgrgid(), ...).  This is good in that it
enables us to not have to do work that clients don't need done (e.g.,
batching up readdir + stat), and it allows us to care a bit less about
round trip optimization.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Jun 2013 11:51:42 -0500

> spaces.  Any suggestions on a better tool chain?

Might lyx2rfc work for you?  https://github.com/nicowilliams/lyx2rfc

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Jun 2013 20:14:50 +0200

> /meeting/(?P\d+).json$'

Did you try applying RFC 6570 URI templates to the description?

Grüße, Carsten

PS.: I have no idea what the ?P in the above is supposed to do, or the trailing single quote.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 20 Jun 2013 09:24:00 -0400

Carsten Bormann <[email-masked]> wrote:
    cb> On Jun 18, 2013, at 03:57, Michael Richardson <[email-masked]>

    >> /meeting/(?P\d+).json$'

    cb> Did you try applying RFC 6570 URI templates to the description?

I'll investigate that.

    cb> PS.: I have no idea what the ?P in the above is supposed to do, or
    cb> the trailing single quote.

trailing quote is copy and paste error.
the ?P is python/django specific, so it would be better to remove it

---

From: Ole Laursen <[email-masked]>
Date: Thu, 20 Jun 2013 17:54:57 +0200

> I agree with adding a test URL, when the test URL is stable and deployed.

Actually what I meant to suggest was that you delete the JSON you've
put in the document and replace it with the test URL. That way it
can't get out of sync which is likely to happen when people touch
these parts in the years to come.

> I don't propose to document anything that isn't self-evident.

Let me just clarify that I'm suggesting you, to the extent possible,
rename attributes that aren't self-evident. For instance, perhaps
"href" should be renamed to "canonical_url".

Ole