At the Forge

People used to say that open-source technologies were excellent at mimicking and copying proprietary projects, but that they had few original ideas of their own. This was never completely true, but I believe that today it is demonstrably false, and that it becomes increasingly false over time. In many respects, the bleeding edge of innovation in the computer industry is taking place within the Open Source community. If you're looking for the future of operating systems, network protocols, Web development or programming techniques, the odds are quite good that you'll find it by looking at open source.

Ruby on Rails, the well-known Web development framework, is such a technology; nearly every other language now has its equivalent of Rails. Rails certainly has been influential on other frameworks in many ways, from its use of MVC to its pervasive “convention over configuration” stance. But, Rails also is leading the way through its use and encouragement of automated testing. Testing has existed for some time in the computer world, but the Ruby community in general and the Rails community in particular are almost fanatical in their desires to test programs extensively, at multiple levels. Such testing, as many others have written over the years, gives you the confidence you need when changing, improving and just refactoring existing code.

For the past few months, I have been looking at a number of testing technologies that came from the Ruby and/or Rails worlds, which I believe are likely to have an impact on other, slower-moving languages and environments. The one thing all these tests have in common is that they are of interest primarily to the programmer. That is, the programmers working on a project all might agree on the need for testing and even on a framework for doing it. The nontechnical managers of a project, although they might benefit from such testing, don't participate in this process very closely, because they typically cannot do so. After all, even Shoulda's syntax, for all its simplicity over standard test/unit constructs, is still in code and, thus, is hard for nonprogrammers to read and understand.

Enter Cucumber, the oddly named but extremely powerful integration testing framework begun by Aslak Hellesoy. Cucumber is part of the BDD (behavior-driven design) school of thought, which argues that development should begin with a specification, and then the code will be written to match that specification.

Typically, in frameworks such as RSpec and Shoulda, the specification is written in code. Cucumber takes a different approach, making it possible to write specifications in English, or other natural languages, and for the computer to take care of translating those specs into an executable format. In this way, the specs still are executable, but they also are readable by nonprogrammers, making it easier to discuss at meetings or in documents. I haven't experienced this firsthand, but it should be possible for nonprogrammers to read the Cucumber stories and begin to execute them.

Cucumber can be used in a variety of testing contexts, but it is most commonly used by Rails programmers for integration tests, and many of the people using it are raving about the effect it has on their development process. This month, I introduce Cucumber, which I believe has the potential to change the ways integrated tests are written and executed dramatically, and which is another clever invention by a member of the Open Source community. I'll walk through the creation of some simple Cucumber tests, and I'll point out where it requires just a bit more ripening. (Vegetable jokes are a staple of the Cucumber community, for better or worse. So, if you plan to use Cucumber, it'll be useful if you find such humor a-peel-ing.)

Cucumber has undergone a number of rapid transformations in the past year alone, thanks in no small part to a growing number of contributors, as well as a great deal of excitement and exposure within the Ruby community. As a result, it is sometimes hard to keep up with version numbers and documentation.

Fortunately, the installation process for Cucumber remains fairly straightforward; it comes packaged as a Ruby gem, which means that you can install it with:

gem install cucumber

At the time of this writing, Cucumber is at version 0.4. Moreover, while Cucumber (and many other Ruby gems) have been hosted by GitHub (a well-known commercial repository for the Git version-control system), it recently was announced that GitHub no longer will support the creation of Ruby gems. So, you might need to look around for Cucumber's official repository when you read this.

Once you have installed Cucumber, you need to add its functionality to your Rails application. You can do this with the following:

script/generate cucumber

This puts the appropriate Rake tasks into place (in lib/tasks/cucumber.rake), adds the initial default step definitions (that is, low-level test implementations) and the overall system support that is necessary for Cucumber to function. All of the files associated with Cucumber are put inside the features directory, which can be somewhat confusing to the uninitiated.

Once these new files are in place, you can run Cucumber as follows:

rake cucumber

Cucumber will run through any file with a .feature suffix in the features directory. If you just installed Cucumber, no such files will exist yet, and you will see output like this:

0 scenarios
0 steps
0m0.000s
Loaded suite /usr/bin/rake
Started

Finished in 0.000232 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

This is similar to the output you would get from running rake test without any tests installed. So let's get started and write something.

Cucumber uses a different vocabulary for creating tests and specifications than you might be used to. Each Cucumber file describes a single “feature” of the application and has a .feature suffix. A feature typically will be a small slice of the application—anything from authentication, to sending or receiving messages, to producing a report. The feature traditionally is described with a line like the following:

Feature: Home page

As you can see, this file begins with the word “Feature:”, and then contains a description. This description, like many others in Cucumber, appears in the output later on, as a helpful description.

Following the Feature declaration, you describe the feature, typically in the form of a “story”, as used in many agile teams and in the BDD world in general. (If you are new to stories, I suggest reading Dan North's blog post on the subject; see Resources.) Here is a typical story:

As a user,
I want to be able to log in
So that I can use the system

The feature is then tested in a number of ways, each of which is known as a scenario. The idea is that each scenario describes a situation (Given) in which the user performs some actions (When), and then sees some results (Then). The scenario should be as specific as possible, testing a particular path through the application's interface.

Note that the scenario is not meant to test one particular controller, model, library or other element of the code. Rather, the scenario should represent a particular action from the user's perspective, which might cover one controller and one model, or a dozen of each. It's normal and reasonable to have a number of scenarios for each feature. It also is reasonable to assume that the number of scenarios will grow over time, as you (and your users) stress the application in new and different ways, and you uncover bugs that need to be covered by new scenarios. A scenario consists of one or more steps, which are translated into working Ruby code. Here is a sample scenario:

Scenario: Get to the login screen

      Given a user named "Reuven" "Lerner" with an e-mail 
      address "reuven@lerner.co.il"
      When I go to the home page
      Then I should see "Web site"
      Then I should see "Login"

I put this scenario (with its story and one feature) into features/login.feature, and then ran rank cucumber. Cucumber responded by going through the file, executing the scenario I had defined. Well, it tried to execute the scenario; here is what I actually saw on the screen:

Feature: Home page

As a user,
I want to be able to log in
So that I can use the system

  Scenario: Get to the login screen      
  # features/login.feature:7
    Given a user named "Reuven" "Lerner" with an e-mail
     ↪address "reuven@lerner.co.il" # features/login.feature:9
      Undefined step: "a user named "Reuven"" (Cucumber::Undefined)
      features/login.feature:9:in `Given a user named "Reuven" "Lerner" 
      with an e-mail address "reuven@lerner.co.il"'
    When I go to the home page     
    # features/step_definitions/webrat_steps.rb:15
    Then I should see "Web site"   
    # features/step_definitions/webrat_steps.rb:123
   Then I should see "Login"       
   # features/step_definitions/webrat_steps.rb:123

1 scenario (1 undefined)
4 steps (3 skipped, 1 undefined)
0m0.012s
You can implement step definitions for undefined steps 
 with these snippets:

Given /^a user named "([^\"]*)" "([^\"]*)" with an 
 ↪e-mail address "([^\"]*)"$/ do
|arg1, arg2, arg3|
  pending
end

rake aborted!
Command failed with status (1): 
 [/System/Library/Frameworks/Ruby.framework/...]

In other words, Cucumber looked for a definition that would handle my step “Given a user”, but did not find one. It stopped interpreting my scenario and threw an error. Cucumber then went further, reminding me that I needed to define this step and giving me an outline for it.

A step definition, as you can see from Cucumber's suggestion, is a regular expression attached to a Ruby block. The regular expression is matched against the Given (or When or Then) statement, with one item matched using parentheses (the standard way of matching in a regular expression), which is then handed to the block as an argument.

Now, let's take the simple step definition and stick it into features/step_definitions/authentication.rb. When rerunning rake cucumber, Cucumber no longer can complain that this step definition is not defined. Rather, it signals that because the step definition is pending, it cannot continue with the rest of the scenario. Let's define this Given step:

Given /^a user named "([^\"]*)" "([^\"]*)" with an 
 ↪e-mail address "([^\"]*)"$/ do
|first_name, last_name, email|
    @person = Person.create(:first_name => first_name,
                            :last_name => last_name,
                            :password => 'password',
                            :email_address => email)
end

You might have noticed that this step definition changed from the original, expecting two quoted words rather than one, with the block taking two parameters rather than one. Let's change the scenario definition so that it contains the step:

Given a user named "Reuven" "Lerner" with an e-mail
 address "reuven@lerner.co.il"

Running this in Cucumber gives the following:

Scenario: Users who go to the home page are asked to log in
# features/login.feature:7
     
Given a user named "Reuven" "Lerner" with an e-mail 
 ↪address "reuven@lerner.co.il" 
 # features/step_definitions/authentication.rb:1
  When I go to the home page   
  # features/step_definitions/webrat_steps.rb:15
  Then I should see "Web site" 
  # features/step_definitions/webrat_steps.rb:123
  And I should see "You must first log in" 
  # features/step_definitions/webrat_steps.rb:123

1 scenario (1 passed)
4 steps (4 passed)
0m0.473s
Loaded suite /usr/bin/rake
Started

Finished in 0.000167 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

If you are wondering who defined the three final steps, look no further than the right-hand side of the output: Webrat, a browser simulator written in Ruby, understands a large number of browser-style step definitions, including “I go to” and “I should see”, allowing you to test for the presence or absence of text in each situation. Cucumber provides a wide variety of Webrat step definitions, such that you can tell Cucumber to go to a page, to fill in a form or to use selection lists, check boxes and radio buttons.

This is basically what it means to work with Cucumber. You create a feature in a .feature file and write one or more scenarios in that .feature file, the lines of which are matched by regular expressions defined in the step_definitions directory. The fact that the .feature file is written in English, from the perspective of the user, means you can show it to nontechnical managers or clients. They even can help write scenarios, and if the scenarios aren't written perfectly for the purposes of Cucumber, they can understand that you are trying to test the application from a variety of perspectives.

It feels a bit strange (at least, it did in my experience) to write scenarios in Cucumber, because you're basically writing code, but in full English sentences. It also took me some time to internalize the fact that each English sentence is similar to a subroutine call, invoking a particular piece of code in the step_definitions directory. Over time, you presumably will create a large library of such step definitions, which you then mix and match within your Cucumber scenarios to test your system.

Here is a second scenario I wrote, in order to test logging in:

Scenario: Users can log in by entering their 
 name and e-mail address
Given a user named "Reuven" "Lerner" with an e-mail
 ↪address "reuven@lerner.co.il"
When I go to the home page
 And I fill in "reuven@lerner.co.il" for "email_address"
 And I fill in "password" for "password"
 And I press "submit"
Then I should see "Welcome back to the site, Reuven!"

Once my two scenarios pass, I commit them to version control and keep them in my application, in the features directory.

If my new scenario doesn't pass, I go through the same iterative process as before—either writing step definitions or fixing bugs in the code to make sure the steps pass. But, Cucumber is a bit slow to execute, so it can be a pain to run through all the features and all the scenarios. So, you can run Cucumber manually, rather than via Rake:

cucumber features/login.feature

You even can indicate that you want to run only the feature starting on line 13 of the file in question:

cucumber features/login.feature:13

This can be a real time-saver when you have a lot of scenarios in a single file and you are trying to debug only one of them.

Cucumber is a well-thought-out system, with a large number of features and abilities that correspond closely with Web developers' needs. First, the step definitions in Cucumber can use either RSpec (the default) or Shoulda, two BDD testing frameworks that have become quite popular in the Rails community.

As I mentioned previously, you can use Cucumber to test models and controllers, and not just for integration testing. My personal preference has been to use Cucumber in this way, however, because it provides a nice, user-side perspective on things and lets you test the site as a user would.

Well, it lets you test the site as a user would, but with one large caveat: Webrat is a great tool, but it doesn't support JavaScript. This means if your site has a great deal of AJAX and JavaScript, you will not be able to test it via Webrat. There are ways to get around this problem, both using Webrat (for example, by including textual links in your application) and by using external testing systems, such as Selenium or Celerity/Culerity. But I have yet to find a system that is easy to integrate, reliable and runs on servers as well as on my desktop.

The fact that Cucumber is slow is a bit of a drawback; it can take quite a while to run through all of the scenarios on a large application. One solution is to use Cucumber's tag feature, which allows you to give one or more tags to a scenario. Then you can run all the scenarios with that tag, across all the features.

If you want to avoid hitting the database each time you create or update (or find) an object, you can integrate a factory (for example, Factory Girl) with Cucumber. This can speed things up, as well as give you a great deal of flexibility in creating scenarios and testing your application.

Cucumber is an innovative approach to testing that has really grown on me and demonstrates the power of English-language, story-based testing to a degree I have yet to see elsewhere. If you are developing in Ruby, I strongly suggest you take a look at integrating Cucumber into your own work.