At the Forge

I often tell the students in my programming classes that back in the 1960s and 1970s, it was enough for a program to run. In the 1980s and 1990s, we needed programs not only to run, but also to run quickly. In the modern era, programs need to run, and run quickly, but even more crucial is that they be maintainable.

Of course, we don't talk much about “programs” any more. Now, we have “applications”, or as Steve Jobs taught us to say, “apps”. This is especially true for Web applications, which are a combination of many different programs, and often different languages, as well—a server-side language, plus SQL, HTML, CSS and JavaScript. And, don't forget the configuration files, which can be in XML, YAML or other formats entirely.

Modern Web frameworks have tried to reduce the potential for clutter and chaos in Web applications. Ruby on Rails was the most prominent framework to suggest that we prefer “convention over configuration”, meaning that developers should sacrifice some freedom in naming conventions and directory locations, if it means easier maintenance. And indeed, when I take over someone else's Rails codebase, the fact that the framework dictates the names and locations of many parts of the program reduces the time it takes for me to understand and begin improving the program.

Even in a Rails application though, we can expect to see many different files and programs. Heroku, a well-known hosting company for Web apps, looked at thousands of apps and tried to extract from them the factors that made it more likely that they would succeed. Their recommendations, written up by then-CTO Adam Wiggins, are known as the “12-factor app”, and they describe practices that Heroku believes will make your app more maintainable and more likely to succeed.

In this article, I take a look at each of the factors of a 12-factor app, describing what they mean and how you can learn from them. I should note that not every aspect of a 12-factor app is unique to this set of recommendations; some of these practices have been advisable for some time. Moreover, it's important to see these as recommendations and food for thought, not religious doctrine. After all, these recommendations come from examining a large number of applications, but that doesn't mean they're a perfect match for your specific needs.

A 12-factor app has “one codebase tracked in revision control, with many deploys”. I remember the days before version control, in which we would modify Web applications on the production server. Obviously, things have improved a great deal since then, and many (most?) developers now understand the importance of keeping their code inside a Git repository.

So, why would it be important to state this as part of a 12-factor app?

It would seem that the reason is two-fold: keep everything you need for the application inside a single repository, and don't use the same repository for more than one app. In other words, there should be a one-to-one correspondence between your app and the repository in which it sits.

Following this advice means, among other things, that you can distribute your app multiple times. I recently finished a project for a client that originally had developed the software in Lotus Notes. Now, I don't know much about Notes, but the fact is that you cannot easily distribute an application written in Notes to new servers, let alone to your laptop. A 12-factor app puts everything inside the repository, dramatically reducing the work needed to deploy to a new server or new environment.

The growth of Vagrant and Docker, two open-source systems that allow for easy virtualization and containers, means that we might see this aspect of the 12-factor app change, looking at “containers” rather than “repositories”. Indeed, Discourse already has moved in this direction, encouraging users to deploy within a Docker container, rather than installing the software themselves. However, the idea would be the same—have one configured version of the application and then deploy it many times.

Every program has dependencies; if nothing else, software depends on the language in which it was written, as well as the core libraries of that language. But if you are using an open-source language and framework, you likely are using numerous packages. A 12-factor app, according to the definition, explicitly declares and isolates dependencies. That is, the application should indicate what external libraries it uses and make it possible to change or remove those dependencies.

This factor does raise the question, at least for me, of “As opposed to what?” In Rails, for example, I cannot easily use a package (Ruby gem) without explicitly mentioning it my application's Gemfile. In Python, I need to import packages explicitly in files that use them. To what is the 12-factor author referring when he says that we shouldn't implicitly use external resources?

The author writes that apps should not “rely on the implicit existence of any system tools”, going so far as to say that they should not “shell out” to run external programs. As a general rule, that's certainly a good idea; the days in which it was acceptable to open a subshell to run external UNIX utilities are long gone. And yet, there are times when it is necessary to do so.

So I have to wonder about the advice given in the 12-factor app, saying that all external programs should be bundled with the application, in its repository. It's a good idea to have everything in one place, but I recently worked on a project that needed to use the open-source PdfTk program. Did I really want to include PdfTk in my app's repository? I expect it wouldn't even work to do that, given the mix of Windows, Macintosh and Linux boxes among the developers, servers and project managers working on the site.

Another aspect of this factor has to do with making not only the library dependencies explicit, but also their versions. It's very easy and tempting simply to use whatever version of a Ruby gem or Python package is installed on a system. But without version numbers, this can cause huge problems—the application might behave differently on various computers, depending on what versions they have installed. Explicitly marking versions that are known to work reduces the chance of such trouble.

It used to be that we would, in our code, explicitly connect to a database or to other servers from within our application. One of the many risks associated with that practice was that if others got a hold of our codebase, they then would have our credentials to log in to databases and third-party services.

A solution has been to put such information in configuration files that aren't committed into the code repository. How do such files then get to the system? That's actually a bit tricky, and there are a number of solutions to the problem.

One solution that has become popular, and is encouraged at 12factor.net, is the use of environment variables. That is, all of the configuration is set in environment variables, in the deployment user's shell or elsewhere in the system (for example, the HTTP server's configuration file). If you don't want to set environment variables yourself, you can use a system like “dotenv” to provide them for you.

But it gets even better. By putting configuration settings in environment variables, you also ensure that you can run the same code in different environments. Want to change the credentials for a third-party service? You don't have to touch or redeploy the code itself. Want to connect to a different database when running your tests? Just change an environment variable.

Of all of the suggestions at 12factor.net, this is the one that I believe offers the most bang for the buck. It increases security and increases the flexibility of your application. The trick is to reduce, as much as possible, the number of “if” statements in your code that test for such values. You want to be using whatever the environment provides, not making conditional statements that change the behavior based on them.

The next factor at 12factor.net says that we should use external services as resources. This is a shift that has been happening more and more. For example, if you ran a Web app on a standalone server ten years ago, you would have just sent e-mail to your customers directly from the app, using the built-in “sendmail” (or equivalent) program.

However, the rise of spam filters and the intricacies of e-mail delivery, as well as the move toward server-oriented architectures (SOA), has encouraged people to move away from using their own systems and toward using separate, third-party systems. Sending e-mail, for example, can be done via Google's Gmail servers or even through a specialized company, such as Sendgrid.

Such third-party services have become increasingly common. This is true because networks have become faster and servers have become more reliable, but also because it's less and less justifiable for a company to spend money on an entire IT staff, when those functions can be outsourced to a third party.

In my opinion, such third-party services are an excellent way to create an application. It allows you to focus on the parts of your application that are special to your organization and not spend time or effort on the configuration and tuning of external services. However, this doesn't mean I'd want to outsource everything; in particular, I'm not sold on the idea of third-party database servers. Perhaps this is something I'll just have to get used to, but for now, there are certain aspects of my apps that I'll continue to keep in-house or at least on servers I've bought or rented.

This factor says you first should build your app and then run it—something that I would think we normally do anyway. Certainly, deployment tools, such as Capistrano, have changed forever the way that I think about deploying apps. As I've started to experiment with such technologies as Chef, Puppet, Vagrant and Docker, I believe it's only a matter of time before we see an app as a self-contained, almost-compiled binary image that we then distribute to one or more servers. I have heard of a growing number of companies that not only use this approach, but that also deploy an entirely new Amazon EC2 server with each new release. If there is a problem with the code on a server, you just shut down the server and replace it with another one.

I'd generally agree that it's a bad idea to modify production code. However, I've certainly been in situations when it was necessary to do so, with particularly hard-to-track-down bugs that even a seemingly identical staging system could not change. Yes, that might point to problems with the testing environment, but when there is a crisis, my clients generally don't want to hear clever suggestions about the development and build process. Rather, they first want to fix the problem and then worry about how they can avoid the same problem in the future.

Nevertheless, if you aren't using a deployment tool to put your sites out on their servers, you might want to consider that.

The next factor says that the application should be one or more stateless processes. The main implication is that the application should be stateless, something I truly would hope has been the case for most Web apps for some time. And yet, when I was speaking with a Fortune 500 company several days ago, asking about the scalability of an application that I'm building for them, they seemed genuinely surprised to hear that we could add as many Web servers as we wanted, because of the “share nothing” architecture.

Now, you do sometimes want to have cached memory or state for users or other resources. In such cases, it's best to use something like Memcached or Redis—or even a full-fledged relational database—to keep the state. This has the advantage of not only keeping it separate from your application, but also of sharing the data across all the Web servers you deploy.

This factor suggests that the application should be a self-contained system and, thus, export itself and its services via HTTP on a particular port. The idea here seems to be that every application should include an HTTP server of some sort and then start that server on a port. In this way, every Web application becomes a resource on the Internet, available via a URL and a port.

I must admit that this strikes me as a bit odd. Do I really want to see my HTTP server as part of my Web app? Probably not, so I don't see a good reason to include them together.

At the same time, I do see a strong advantage of considering each Web application as a small SOA-style resource, which can be queried and used from other applications. The entire Web is becoming an operating system, and the API calls of that operating system are growing all the time, as new URLs expose new services. By exposing your application as a resource, you are contributing to this new operating system and increasing the richness of the apps that run on it. However, I'm not convinced that where the HTTP server lies, and how closely it is bound to the app, really affects that mindset.

Those of us in the Linux world are fully familiar with the idea of creating new processes to take care of tasks. UNIX has a long history and tradition of using processes for multitasking, and while threads certainly exist, they're not the default way of scaling up.

The “concurrency” section of 12factor.net says that we should indeed use processes and not be afraid to spin up processes that will handle different aspects of our application. Each process then can run a specialized program, which communicates with the other processes using an API—be it a named pipe, socket, a database or even just the filesystem.

True, we could use threads for some of these things. But as 12factor.net says, threads cannot be moved to a different server or VM, whereas processes (especially if they don't contain state and save things to a common storage facility) can.

This aspect of 12factor.net says that we can start up or shut down an app at any time, on any number of servers. To be honest, I'm not sure how this is different from the existing world of Web applications. For as long as I can remember, I was able to start up a Web app without too much fuss and take it down with even less trouble.

“Fast startup” is a good goal to have, but it can be hard to achieve, particularly with the overhead of many modern frameworks and languages. If you're using a language that sits on top of the Java virtual machine (JVM), you're likely to experience slow startup time but very fast execution time.

That said, I'm not sure how important it is to be able to start up a new copy of your application quickly, relative to other issues and constraints. True, it's frustrating to have slow startup times, particularly if those affect your ability to run a test suite. But most of the time, your application will be running, not starting up—thus, I'd downplay the importance of this particular factor.

The idea behind this factor is extremely simple but also important: keep your development, staging and production environments as similar as possible.

It's hard to exaggerate the number of times I have experienced problems because of this. The system worked one way on my own development machine, another way on another programmer's machine, a third way on the staging server and a fourth way on the production machine. Such a situation is asking for trouble. It means that even if you have excellent test coverage of your application, you're likely to experience hard-to-debug problems that have to do with the configuration or the inherent differences between operating-system versions.

Once again, an increasingly popular solution to this problem is to use a virtual machine and/or a container. With a Vagrant VM, for example, you can share the same machine, not just the same environment, among all developers and servers. Working in this way saves time and reliability, although it does admittedly have some negative effect on the performance of the system.

I love logs and sending data to them. In a small application, it's enough to have a single logfile on the filesystem. But if you're using a read-only system (such as Heroku), or if you are going to create and remove servers on a regular basis with Chef or Puppet, or if you have multiple servers, you will likely want to have logs as an external service.

Now, old-time UNIX users might say that syslog is a good solution for this. And indeed, syslog is fairly flexible and allows you to use one system as the logging server, with the others acting as clients.

The 12-factor suggestion is to go one step further than this, treating a log as a writable event stream to which you send all of your data. Where does it go? It might be syslog, but it's more likely going to be a third-party service, which will allow you to search and filter through the logs more easily than would be possible in simple text files.

I must admit there's still come comfort in my being able to run a tail -f on a textual logfile or grep on a file that's of interest to me. But I have used some third-party logging solutions, such as Papertrail, and have come away impressed. There also are open-source solutions, such as Greylog2, which some of my clients have used to great satisfaction.

The final factor in a 12-factor app is that of administrative processes. Now, I often compare a Web app to a hotel, in that the end user sees only the minority of the actual workings. Just as guests never see the kitchen, laundry or administrative offices of a hotel, users of a Web app never see the administrative pages, which often can be extensive, powerful and important.

However, the 12-factor app prescription for admin processes isn't about administrative parts of the site. Rather, it's about the administrative tasks that you need to do, such as updating the database. This factor says that you really should have a REPL (that is, a read-eval-print loop, aka an interactive shell), and that you can put many administrative tasks into small programs that execute.

I agree that a REPL is one of the most powerful and important aspects of the languages I typically use. And I love database migrations, as opposed to manual tinkering with the database that always can lead to problems. However, I'm not sure if this warrants inclusion as a crucial component of a maintainable Web application.

I see the 12-factor app as a great way to think about Web applications and often to increase their stability and maintainability. In some ways, I see it as similar to design patterns, in that we have gained a language that allows us to communicate with others about our technical design in a way that all can agree on. However, as with design patterns, it's important to see this as a tool, not a religion. Consider your needs, take the 12-factor app prescriptions into account, and apply as necessary. If all goes well, your app will end up being more scalable, reliable and maintainable.