At the Forge

In my last few articles, I've considered Web application performance in a number of different ways. What are the different parts of a Web application? How might each be slow? What are the different types of slowness for which you can (and should) check? How much load can a given server (or collection of servers) handle?

So in this article, I survey several open-source tools you can use to better identify how slow your Web applications might be running, in a number of different ways. I should add that as the Web has grown in size and scope, the number and types of ways you can check your apps' speed also have become highly diverse, such that talking about “load testing” or “performance testing” should beg the question, “Which kind of testing are you talking about?”

I also should note that although I have tried to cover a number of the most popular and best-known tools, there are dozens (and perhaps hundreds) of additional tools that undoubtedly are useful. If I've neglected an excellent tool that you think will help others, please feel free to send me an e-mail or a Tweet; if readers suggest enough such tools, I'll be happy to follow up with an additional column on the subject.

In my next article, I'll conclude this series by looking at tools and techniques you can use to identify and solve client-side problems.

One of the problems with load testing is that it often fails to catch the problems you experience in the wild. For this reason, some of the best tools that you have at your disposal are the logfiles on your Web server and in your database. I'm a bit crazy about logfiles, in that I enjoy having more information than I'll really need written in there, just in case. Does that tend to make my applications perform a bit worse and use up more disk space? Absolutely—but I've often found that when users have problems, I'm able to understand what happened better, and why it happened, thanks to the logfiles.

This is true in the case of application performance, as well. Regarding Ruby on Rails, for example, the logfile will tell you how long each HTTP request took to be served, breaking that down further into how much time was spent in the database and creating the HTML output (“view”). This doesn't mean you can avoid digging deeper in many cases, but it does allow you to look through the logfile and get a basic sense of how long different queries are taking and understand where you should focus your efforts.

In the case of databases, logfiles are also worth a huge amount. In particular, you'll want to turn on your database's system that logs queries that take longer than a certain threshold. MySQL has the “slow query log”, and PostgreSQL has the log_min_duration_statement configuration option. In the case of PostgreSQL, you can set log_min_duration_statement to be any number of ms you like, enabling you to see, in the database's log, any query that takes longer than (for example) 500 ms. I often set this number to be 200 or 300 ms when I first work on an application, and then reduce it as I optimize the database, allowing me to find only those that are truly taking a long time.

It's true that logfiles aren't quite part of load testing, but they are an invaluable part of any analysis you might perform, in production or even in your load tests. Indeed, when you run the load tests, you'll need to understand and determine where the problems and bottlenecks are. Being able to look at (and understand) the logs will give you an edge in such analysis.

Once you've set up your logfiles, you are ready to begin some basic load testing. Apachebench (ab) is one of the oldest load-testing programs, coming with the source code for Apache httpd. It's not the smartest or the most flexible, but ab is so easy to use that it's almost certainly worth trying it for some basic tests.

ab takes a number of different options, but the most useful ones are as follows:

Thus, if I want to start testing the load on a system, I can say:

ab -n 10000 -c 100 -i http://myserver.example.com/

Note that if you're requesting the home page from an HTTP server, you need to have the trailing slash, or ab will pretend it didn't see a URL. As it runs, ab will produce output as it passes every 10% milestone.

ab produces a table full of useful information when you run it. Here are some parts that I got from running it against an admittedly small, slow box:

Concurrency Level:      100
Time taken for tests:   36.938 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1118000 bytes
HTML transferred:       0 bytes
Requests per second:    27.07 [#/sec] (mean)
Time per request:       3693.795 [ms] (mean)
Time per request:       36.938 [ms] (mean, across all concurrent
                         ↪requests)
Transfer rate:          29.56 [Kbytes/sec] received

In other words, my piddling Web server was able to handle all 1,000 requests. But it was able to handle only 27 simultaneous requests, meaning that about 75% of the concurrent requests sent to my box were being ignored. It took 3.6 seconds, on average, to respond to each request, which was also pretty sad and slow.

Just from these results, you can imagine that this box needs to be running more copies of Apache (more processes or threads, depending on the configuration), just to handle a larger number of incoming requests. You also can imagine that I need to check it to see why going to the home page of this site takes so long. Perhaps the database hasn't been configured or optimized, or perhaps the home page contains a huge amount of server-side code that could be optimized away.

Now, it's tempting to raise the concurrency level (-c option) to something really large, but if you're running a standard Linux box, you'll find that your system quickly runs out of file descriptors. In such cases, you either can reconfigure your system or you can use Bees with Machine Guns, described below.

So, what's wrong with ab? Nothing in particular, other than the fact that you're dealing with a simple HTTP request. True, using ab's various options, you can pass an HTTP authentication string (user name and password), set cookies (names and values), and even send POST and PUT requests whose inputs come from specified files. But if you're looking to check the timing and performance of a set of user actions, rather than a single URL request, ab isn't going to be enough for you.

That said, given that the Web is stateless, and that you're likely to be focusing on a few particular URLs that might be causing problems, ab still might be sufficient for your needs, assuming that you can set the authentication and cookies appropriately.

The above also fails to take into account how users perceive the speed of your Web site. ab measured only the time it took to do all of the server-side processing. Assuming that network latency is zero and that JavaScript executes infinitely fast, you don't need to worry about such things. But of course, this is the real world, which means that client-side operations are no less important, as you'll see in my next article.

If there's an award for best open-source project name, I think that it must go to Bees with Machine Guns. Just saying this project's name is almost guaranteed to get me to laugh out loud. And yet, it does something very serious, in a very clever way. It allows you to orchestrate a distributed denial-of-service (DDOS) attack against your own servers.

The documentation for BWMG states this, but I'll add to the warnings. This tool has the potential to be used for evil, in that you can very easily set up a DDOS attack against any site you wish on the Internet. I have to imagine that you'll get caught pretty quickly if you do so, given that BWMG uses Amazon's EC2 cloud servers, which ties the servers you use to your name and credit card. But even if you won't get caught, you really shouldn't do this to a site that's not your own.

In any event, Bees assumes that you have an account with Amazon. It's written in Python, and as such, it can be installed with the pip command:

pip install beeswithmachineguns

The basic idea of Bees is that it fires up a (user-configurable) number of EC2 machines. It then makes a number of HTTP requests, similar to ab, from each of those machines. You then power down the EC2 machines and get your results.

In order for this to work, you'll need at least one AWS keypair (.pem file), which Bees will look for (by default) in your personal ~/.ssh directory. You can, of course, put it elsewhere. Bees relies on Boto, a Python package that allows for automated work with AWS, so you'll also need to define a ~/.boto file containing your AWS key and secret (that is, user name and password).

Once you have the keypair and .boto files in place, you then can set up your Bees test. I strongly suggest that you put this in a shell script, thus ensuring that everything runs. You really don't want to fire up a bunch of EC2 machines with the bees up command, only to discover the following month that you forgot to turn it off.

Bees uses the bees command for everything, so every line of your script will start with the word bees. Some of the commands you can issue include the following:

So, if you want to do the same thing as before (that is, 1,000 requests), but now divided across ten different servers, you would say:

bees up -s 10 -g beesgroup -k beespair
bees attack -n 100 -c 10 -u http://myserver.example.com/
bees down

When you run Bees, the fun really begins. You get a verbose printout indicating that bees are joining the swarm, that they're attacking (bang bang!) and that they're done (“offensive complete”).

The report at the conclusion of this attack, similar to ab, will indicate whether all of the HTTP requests were completed successfully, how many requests the server could handle per second, and how long it took to respond to various proportions of bees attacking.

Bees is a fantastic tool and can be used in at least two different ways. First, you can use it to double-check that your server will handle a particular load. For example, if you know that you're likely to get 100,000 concurrent requests across your server farm, you can use Bees to load that up on 1,000 different EC2 machines.

But another way to use Bees, or any load-testing tool, is to probe the limits of your system—that is, to overwhelm your server intentionally, to find out how many simultaneous requests it can take before failing over. This simply might be to understand the limits of the application's current architecture and implementation, or it might provide you with insights into which parts of the application will fail first, so that you can address those issues. Regardless, in this scenario, you run your load-testing tool at repeatedly higher levels of concurrency until the system breaks—at which point you try to identify what broke, improve it and then overwhelm your server once again.

A possible alternative to Bees with Machine Guns, which I have played with but never used in production, is Locust. Locust can run on a single machine (like ab) or on multiple machines, in a distributed fashion (like Bees). It's configured using Python and provides a Web-based monitoring interface that allows you to see the current progress and state of the requests. Locust uses Python objects, and it allows you to write Python functions that execute HTTP requests and then chain them together for complex interactions with a site.

If you're interested in testing your servers, there are several high-quality, open-source tools at your disposal. Here, I looked at several systems for exploring your server's limits, and also how you can configure your database to log when it has problems. You're likely going to want to use multiple tools to test your system, since each exposes a different set of potential problems.

In my next article, I'll look at a variety of tools that let you identify problems and slowness within the client side of your Web application.