At the Forge

The Web was designed for people. When Tim Berners-Lee created the trio of standards that make up the Web—HTTP, HTML and URLs—the intention was for people to browse Web sites, submit information to them and be at the heart of the experience. But for some time now, the notion of the Web as a set of sites that people browse has been somewhat untrue. True, hundreds of millions of people visit an equally large number of sites each day; however, more and more of the visitors to sites aren't people, but programs.

Some of those programs exist to collect information on behalf of other systems. For example, when you search the Web using a site such as Google, you're obviously not searching through all of these sites in real time. Rather, you're asking Google to search through its massive index—an index that it has created and updated via its “bots”, programs that go to a Web site, pretend to browse as a person, and then track whatever they find.

But more and more, the programs that are visiting sites aren't doing it on behalf of search indexes. Rather, they're doing it on behalf of...well, on behalf of themselves. Computers exchange information via the Web, using a variety of protocols and data formats. Native apps on mobile devices are using the Web behind the scenes to query Web applications. And, even those Web applications using Ajax are interacting with a Web site without directly being asked to do so.

This represents a massive shift in what Web applications are doing. No longer are we just producing HTML for users (and search bots). Now, we're producing output that's meant for programmatic consumption—and in many cases, the same people are writing the client and server sides. Sure, we could use “scraping” techniques to retrieve the HTML and search through it, but why do so? If we already know we'll be sending data to a program, there's no reason to send HTML. Rather, we can send it in a more program-friendly data format, without all the bells and whistles that people require.

When such use began, people made a big deal out of it. This trend was known as “Web services”, and a lot of companies—most prominently Amazon—jumped on them, describing all sorts of standards, from XML-RPC to SOAP to WSDL. These protocols are still used, especially by large-scale enterprise applications, to communicate with one another.

But during the last few years, a more informal sort of API has emerged. Sometimes it's based on XML, but more often it's based on JSON, the “JavaScript Object Notation” format that works not only with JavaScript, but with a wide variety of other languages as well.

(By “more informal”, I don't mean that it's not useful or that more formality is needed. I'm merely referring to the fact that it requires coordination between the client and server software authors, rather than adherence to a specification document or established protocol.)

This month, I'm looking at these sorts of APIs—why you would want them, the different styles you can use in designing them, and then how to access and use them.

If you're running a Web application, you probably will want to provide an API at some point. Why? Well, there are a number of reasons:

I'm sure there are additional reasons for providing an API, but even one of these might be a compelling reason in your case—and two or three of them might apply as well. There's also an element of customer openness and trust that comes with an API. I'm much more likely to use a Web application, particularly if it's one for which I'm paying, that provides an API to some or all of its functionality. Even if I never end up making use of it, I know I can do so potentially, which gives me a feeling of being a potential partner of the application's authors, rather than a simple user.

The above also demonstrates that even if you never plan to open up your API to the outside world, it might still be in your interest to design one. Indeed, I've recently heard several experienced Web developers argue that a modern Web site should not be designed as a set of pages, with the API tacked on as an afterthought, but rather as a set of APIs, which can be accessed via a mobile app, a remote client application, Ajax calls or a client-side framework, such as Backbone. In other words, first you establish your APIs, and then you get to work writing applications that use those APIs.

In many ways, this is an attractive thought, one that has the potential to make applications cleaner and easier to write and test. After all, the idea behind the MVC (model-view-controller) paradigm is to separate the different components, such that the business logic has no interaction with the interface presented to the user. MVC-style Web frameworks, such as Rails and Django, encourage this separation, but creating an API makes the distinctions even sharper.

If you have decided to create an API, there are several questions to ask. One of them is what style of API you will use. The Ruby on Rails community has rallied around the idea of REST—“representational state transfer”, a term coined by Roy Fielding—which basically assumes that each URL uniquely identifies a resource on the Internet. Different actions performed on that resource do not make use of different URLs, but rather of different HTTP request methods.

For example, if you have an address book containing information on many people, the first entry might have a URL of:

/people/1

In such a case, you could retrieve information with:

GET /people/1

and a new entry with:

POST /people

Remember that POST requests send their name-value pairs separately from the URL. Because the parameters aren't part of the URL, it sometimes can be a bit tricky to know precisely what is being sent to the server. I generally check the parameters that are being sent using a combination of tools, including the server's logfile, the Firebug plugin for Firefox, the Web developer plugin for Firefox or the ngrep command-line tool for finding and displaying selected network packets.

In the Ruby on Rails universe, you can update an existing entry using the little-known (and little-supported, at least for now) PUT request method:

PUT /people/1

As with POST, the parameters in a PUT request are sent separately from the URL. Trickier yet is the fact that many browsers cannot handle PUT requests directly. The solution that Rails employs, for the time being, is to use POST, but to add a “_method” parameter as part of the request. When the server sees this, it uses the action that should be associated with PUT, rather than with POST. The system works well, although it obviously would be better if browsers were able to support all of the standard HTTP requests.

One of the key things to remember when working with REST is that URLs should refer to nouns, not verbs. Thus, it's perfectly acceptable, within the REST world, to have URLs that refer to any object on your system, from users to books to airplanes to credit-card statements. However, it's not acceptable to name the action you wish to take in the URL. So:

/airplanes/523

would be perfectly acceptable, but:

/airplanes/get_passenger_list/523

would not be. The difference being, of course, that get_passenger_list is presumably the name of an action that you wish to take on the airplane resource. This means you're no longer using the URL to refer to a specific resource, but instead to an action.

When it was announced that Rails was moving toward REST, I must admit I was somewhat resistant. I preferred to use URLs that had been traditional in the Rails world until then, naming the controller and action, as well as an object ID, in the URL. Thus, if I wanted to retrieve one person's address, I would use a URL such as:

/people/get_address/2341

where 2341 would be the unique ID for that person. And, for the most part, this paradigm worked just fine.

But, then I discovered what many Rails developers find out: that Rails is, as it claims to be, “opinionated” software, meaning that things are extremely easy if you do them the way it was designed, and very difficult if you try it in any other way. And as time went on, it became clear that my non-REST URLs were giving me problems. Many elements of Rails were no longer working for me, because they depended on REST. When I started to work with Backbone.js in 2011, I found that Backbone works with a variety of back ends, including Rails, but that using a non-REST interface was clumsy to work with, if it was even possible to use at all.

Thus, I began to get REST religion and tried to make every application I wrote conform to this sort of API, and it worked in many ways. The APIs were consistent, and my code was consistent. Because I was using the scaffolding that Rails provided—if only at the start of a project—the code was smaller and more consistent than otherwise would have been the case. In the case of Rails, which dynamically creates identifiers that represent specific URLs (for example, addresses), sticking with the RESTful routes really simplified things.

That is, until it didn't do so. REST, at least in the Rails implementation, gives you a single HTTP request method for each action you want to perform on your resource. This works well, in my experience, until you want to retrieve resources based on parameters, at which point things can get a bit complicated. Sure, you can pass parameters in the HTTP request, but at a certain point, I personally would rather have several small methods than one method with a huge set of if-then statements reflecting different combinations of parameters.

Thus, I've pulled back a bit from my REST absolutism, and I feel like I've reached something of a balance. For creation, updating and deletion, I'm totally fine with using the RESTful paradigm. But when it comes to retrieving resources from a Web application, I've relaxed my requirements, trying to make my API be as expressive and flexible as possible, without creating a huge number of routes or actions in my controller.

For example, I'm totally fine with retrieving information about a single user using the /users URL, tacking on an ID number to get information about a specific one. But, I often want to implement a search system to look for people in the system whose names match a particular pattern. Back in my pre-REST days, I would have used a search controller and had one or more methods that performed a search. In my neo-REST world, I simply add a “search” method to my “users” resource, such that the URL /users/search represents a search. Is that breaking REST? Absolutely. But, I've found that it makes my API more understandable and maintainable for my users, as well as for myself.

Rails itself, as opinionated and RESTful as it might be, offers you an out in the routes file (config/routes.rb). A recent project on which I worked had the following route:

resources :articles

This translates into the usual combination of RESTful routes and HTTP request methods. But when I decided to add three additional API calls to grab particular types of articles, I was able to do this by adding a block to the resources declaration:

resources :articles do
    get :latest, :on => :collection
    get :article_links, :on => :member
    get :stories, :on => :collection
end

Not only does Rails offer this possibility, but it differentiates between “member” routes (which require a resource ID) and “collection” routes (which don't require a resource ID, because they operate on the entire collection).

Once you've set up your API-naming convention, you need to consider what data you'll be passing. This means deciding on a data format, the parameters you'll receive in that format and the response you'll send back in that format.

When it comes to formats, there are really two main players nowadays: XML and JSON. As I mentioned previously, XML is very popular among enterprise users, but JSON has become very popular because of how easily you can transform objects from such languages as Python and Ruby into JSON (and back). In addition, JSON is nearly as self-documenting as XML, without the huge textual overhead or the complexity of an XML parser. Like many other people, I've switched to JSON for all of my API needs, and I haven't regretted it at all.

That said, Rails offers the option of responding in any of several different formats with the respond_to block. It basically lets you say, “if users want JSON, do A, but if they want XML, then do B.”

As for the request and response parameters, I try to keep it pretty simple. However, there's one parameter you absolutely should include in any API you create, and that's a version number. Your API presumably will evolve and improve over time, adding and removing method names, parameters and parameter types. By passing a version number along with your parameters, you can ensure that you're getting the parameters you require, and that both the client's and server's expectations are in sync.

Moreover, I've found that you can use that version number as a key in a hash of parameter lists. That is, rather than having separate variables in your server-side program that track which parameters are expected for each version, you can have a single hash whose keys are version numbers and whose values are arrays of expected parameters. You even can go one step further than this, having a multilevel hash that tracks different parameters for various API calls. For example, consider the following:

EXPECTED_PARAMS = { 'location' => {
    1 => ['longitude', 'latitude', 'altitude'],
    2 => [longitude', 'latitude', 'altitude', 'speed', 'timestamp'],
  },

  'reading' => {
    1 => ['time', 'area', 'mean', 'good', 'number'],
    2 => ['time', 'area', 'mean', 'good', 'number', 'seen', 'span',
    ↪'stdDev']
  }
}

Then, you can do something like the following:

version_number = params['version_number'].to_i
method_name = params['action']
required_fields = EXPECTED_PARAMS[method_name][version_number]

This is far easier than a lot of if-then statements, and it allows me to centralize the definition of what I expect to receive from my API clients. If I wanted to validate the data further, I could make each array element a hash rather than a string, listing one or more criteria that the data would need to pass.

Finally, the response that you give to your API users is your public face to the world. If you provide useless error messages, such as “Something went wrong”, you'll presumably discover that developers are less than happy to use your system or develop on top of it. But if you provide detailed responses when things go well and poorly, not only will developers enjoy your system, but their end users will as well.