At the Forge

This might come as a surprise, given that I have spent much of my professional career working with, writing on and teaching about the use of databases, but there was a period (mostly during and right after college) when I really didn't understand why people ever would need them. After all, I thought, you can just store and retrieve information in a file on your disk, no? My attitude back then demonstrated not only profound ignorance about databases themselves, but also about the types of problems people need to solve and the ways in which database technology had, even then, been developed to solve those problems.

Now I'm not quite as dismissive of technologies as I was back in my college days. But, it's true that although I've long heard of “message queues”, it's been only in the last year or so, while working on a project, that I've come to realize just what a useful innovation they are. Sure enough, now that I understand how and why I would want to use them, I see uses for message queues everywhere.

In this article, I introduce the idea of message queues and give several examples of how you can install and use them. I also discuss why you might want to use a message queue, particularly on a Web application, which people typically think of as consisting only of an HTTP server and related software.

If you've been programming for any period of time, you know that a fundamental data structure is the queue, or FIFO (first in, first out). Like a queue at the post office, the first item stored also is the first item removed. Different queue implementations have different capabilities, but the general idea is that you put something in the queue and then retrieve it when it's ready. In Ruby (and many other languages, such as Python), you can implement a queue as follows:

class Queue
  def initialize
    @queue = [ ]
  end

  def enqueue(thing)
    @queue << thing
  end

  def dequeue
    @queue.shift
  end

  def to_s
    @queue.join(', ')
  end
end

You then can use it as follows:

>> require 'queue' #=> true
>> q = Queue.new #=>
>> q.enqueue('a') #=> ["a"]
>> q.enqueue('b') #=> ["a", "b"]
>> q.enqueue('c') #=> ["a", "b", "c"]
>> q.dequeue #=> "a"

Because queues in Ruby can hold any value, you don't need to worry about what will be on the queue. You just know that you can stick whatever you want on there, and that eventually you can retrieve it, in order.

(I should note that if you're actively programming in Ruby, I hope you're not really using a queue class like this, but that you're rather just using arrays, which support all the basic operations you're likely to need to work with simple queue data structures.)

Queues are great, but I'm sure you can already imagine all sorts of horrible scenarios if you were to use this simple one for something important, such as a list of bank transfers to execute. My banker's desk is full of piles of papers that she needs to handle, and she (presumably) works through them from top to bottom, dealing with each one in turn, but it would be pretty unforgivable for one or more of those papers to get lost. And, although it's easy to say that Ruby arrays are pretty stable, what happens if the power goes out? In such a case, the entire queue is lost, causing untold problems for the people who expected safe delivery.

The difference between the simple-minded queue I showed above and a true message queue is that the latter guarantees delivery of every message. This means that basically no matter what happens, you can be sure the message eventually will be delivered, despite power outages and other issues. But, message queues are even better than that. Not only do they guarantee delivery, but they also work quickly, allowing you to queue up a number of messages or actions that require attention, but for which you lack the resources to handle immediately.

For example, consider a Web application that is designed not to provide immediate feedback to users, but rather to receive and process information sent from other computers or mobile devices. This type of application typically doesn't require giving the user immediate feedback (other than an acknowledgement that data was received). All of the messages sent are of great importance (and should not be lost), but the number of messages can vary greatly from minute to minute, let alone from hour to hour. When the data is processed and eventually placed in the database, however, doesn't matter nearly as much.

There are more mundane examples, as well. Consider a Web application that needs to send e-mail updates to people, such as from a calendaring application. If the application were to send e-mail each and every time an event were changed, the response time—or the number of server processes available to receive new incoming messages—might well suffer. Instead, the application can stick the mail-sending task on a message queue and then let a process on a separate computer retrieve the order and send the actual e-mail.

Offloading the retrieval of messages to a separate computer offers another performance advantage. It allows you to scale up the processing as necessary, by adding additional back-end computers. Given that a message queue is transactional (that is, all-or-nothing), you can have as many back-end machines retrieving from the queue as you want. You don't have to worry that the same message will be delivered twice or that two processes will have to fight over the retrieved data.

So, now that I've convinced you that you want to have a message queue, how do you go about using one? The first question is which one to choose. Many message queues exist, and each has its advantages and disadvantages. I've been using Amazon's Simple Queue Service on a project for the past number of months, and although it certainly has its downsides—it costs money, and it can take a bit of time for messages to percolate through the system—the advantages have been fairly clear, including Amazon's willingness to store messages for up to two weeks and its impressive uptime statistics. And, although I certainly could have set up my own message-queueing system, I've been working on other aspects of the project and appreciated that someone else, out there in “the cloud”, was dealing with the various IT-related tasks associated with running a queue.

If you have used any of Amazon's previous cloud offerings, its queue service will not be a surprise. You need to have an Amazon account and sign up for a unique access key that will identify you to Amazon for identification and billing purposes. In addition to the access key, which you can think of as a user name, you also need a secret (akin to a password), which is sent to Amazon along with each request.

Once you have set yourself up with SQS, you need to connect to it, preferably (but not necessarily) using one of the many SQS client libraries that have been developed. Most of my work nowadays is in Ruby, and when I started my project, I found that the best-known Ruby gem for SQS access was from RightScale, in the “right_aws” package. I have been using this driver without any problems, but it's true that Amazon has since released its own drivers for Ruby. I hope to experiment with that driver in the near future and to compare it with the RightAws modules—although to be honest, I don't expect to see any significant differences.

If you're using another language, there almost certainly are libraries you can use as well. For the Python community, there are the boto packages. See Resources for more information.

By the way, it's true that SQS costs money. However, queueing systems exist to handle large quantities of data, which means they're going to charge you very little per message. How little? Well, according to the prices posted at the time of this writing, sending messages is absolutely free. Receiving messages is free for the first GB each month. After that, you pay nothing for the first GB, and then 12 US cents for each GB, up to 1TB. Now, Amazon does have a number of different server centers, and each might have its own pricing. Also, pricing is applicable only when going in or out of Amazon's server systems. This means if you're using a hosting solution, such as Heroku, which sits on Amazon servers, transfer to and from SQS is completely free. Actually, that isn't quite true—data transfers are free only if you stay within the same geographic server cluster. My point, however, is that for most people and projects, the pricing should not be an issue.

I'm using SQS for a Web application that is (by the time you read this, if all goes well) intended to receive JSON data from mobile devices, sent via an HTTP POST request. The JSON data then needs to be parsed and stuck into a relational database, but that doesn't need to happen right away. The architecture of the application, thus, consists of two separate parts. The main Web app receives the data and puts it onto the message queue with minimal parsing and validation. A separate Web app, running on a separate server, retrieves the JSON data, parses and validates it, and then puts it into the database. From the perspective of SQS, the fact that I'm using different servers really doesn't matter at all; as long as I connect to SQS with the right user name and password, and use the right queue name for sending and receiving, everything will be just fine.

Before you can send to or receive from SQS, you first must connect to it. Since I'm using the right_aws gem for Ruby, I need to download and install that:

$ gem install right_aws

Note that because I'm using rvm, the Ruby version manager, I installed this gem as my own user. If I were installing it for the entire system, or if I were not using RVM, I would need to log in as root or use sudo to execute the command as root.

With the right_aws gem installed and in place, I now can use it to connect to the SQS server. Note that RightScale's gem provides access to several different APIs, including several different “generations” of SQS. I am using the second-generation API, via the RightAws::SqsGen2 class.

I've put my Amazon keys in a separate YAML-formatted configuration file, allowing me to change and update keys as necessary, as well as keep track of separate keys for different environments. I then read the configuration information into my program with the following line:

SQS_CONFIG = YAML.load_file("/Users/reuven/
↪Desktop/config.yml")['defaults']

The above takes each of the name-value pairs in the “defaults” section of my config.yml file and puts it into a hash named SQS_CONFIG. (Note that I've used all caps to indicate that this is a constant and should not be modified by other programmers unless they have a really, really good reason for doing so.)

I then can get a connection to SQS with the following code:

require 'right_aws'
sqs = RightAws::SqsGen2.new(SQS_CONFIG['aws_access_key_id'],
                            SQS_CONFIG['aws_secret_access_key'],
                            { :server => SQS_CONFIG['sqs_server'] }
                            )

As you can see from the above call, Right::SqsGen2.new takes three parameters: the AWS key, the AWS secret and a hash of options that help configure the queue object. The most important one to pass is the name of the SQS server you want to use. If you don't specify it, you'll get queue.amazonaws.com, but to be honest, I haven't really thought about it much since checking with Heroku (our hosting provider) about which server to use.

Once you're connected to SQS, you must create (or retrieve) a queue. You can think of a queue as a single array to which you can store or retrieve data, just as I did in my simple Queue class earlier in this article. The difference, of course, is that the actual data storage is happening across the network, on servers to which you have no direct access. You can have any number of queues, each with its own name, containing alphanumeric characters, hyphens and underscores. So, if you want to use a queue called “testq”, just say:

sqs_queue = sqs.queue('testq')

This returns an instance of RightAws::SqsGen2::Queue, an object that represents an Amazon message queue. A number of methods are defined on this object, including creation (which I do via the above call, rather than directly), deletion (which will remove all of your data, so I really wouldn't suggest it unless you have to), and the sending and receiving of messages. You also can set the visibility timeout on this object, which tells Amazon how long a message should be invisible once it has been read, but before it has been deleted. You even can get the size of the message queue, using the size method.

In my simple, non-distributed message queue example, you saw that new messages are added to the queue using an enqueue method, taking a single object as a parameter. The same is true in this case; if you want to send a message to the queue, you simply say:

my_message = 'hello!'
sqs_queue.send_message(my_message)

This will turn your string into an SQS message and send it to the queue. So long as the message is less than 64KB in size and is in text format (including JSON or XML), Amazon probably will accept it. (The RightScale gem claims to support messages only up to 8KB in size, just as Amazon used to do, but it's not clear to me whether the gem enforces these limits or if Amazon's updates are reflected by the gem's behavior.) Trying to send a message that's too long for Amazon's limits will result in an exception being thrown. There is an explicit list on the SQS FAQ page of which UTF-8 characters are acceptable in an SQS message.

One nice thing about SQS is that you can have any number of messages in a queue at a time; there is no defined limit. By default, messages are kept in a queue for four days, but you can configure that to be anywhere from one hour to two weeks.

So, you've sent a message to the message queue. How do you receive it? After going through the initial configuration, connection and queue creation/opening displayed above, you can retrieve the first waiting message on the queue with:

message = mothra_queue.receive

In RightScale's Ruby library, message is set to nil if no messages were available. Thus, before you can operate on the message, you must first check to ensure that it's non-nil.

Assuming that message is not nil, you can get contents by transforming the message into a string—in other words, by invoking .to_s on the message:

print message.to_s

When you retrieve a message, Amazon keeps a note of that and makes it invisible to other processes that might try to retrieve it. In other words, if you've queued a single message and then retrieve that message, other processes trying to retrieve from the queue will be told that no messages are available. However, this is true only for a short time. Once the visibility timeout has passed, the message is once again available to retrieving processes. So, in order to ensure that a message is not read twice, it must be deleted:

message.delete

Under most circumstances, you will want to retrieve and then delete a message almost right away.

If you're saying, “Well, that seems quite simple”, you're right. Message queues are a dead-simple idea, particularly if you're familiar with queues as data structures. Distributed message queues can be quite difficult to get to work in a distributed and persistent way, but Amazon has done just that and makes its queue available for a very reasonable price, often ending up free for small organizations and sites.

The advantages that a distributed message queue can bring to the table are overwhelming though, particularly when you have tasks or pieces of data that are coming in too rapidly to handle, but which could be processed by a large number of back ends. It's easy to imagine a large number of back-end computers picking messages off and inserting them into a database, after parsing and checking them for validity. Indeed, that's what I'm doing on my current project, and it has been working like a charm.

Now, there are issues with Amazon's queues. For starters, they have longer latency than you would get with a local queue, and they also are sitting on third-party servers, which might not sit well with some companies. But for the most part, it has worked without a hitch and has become a core part of the infrastructure on my project. During the course of this work, I've started to find all sorts of uses for message queues, and I'm starting to incorporate them into other projects on which I work. The day may come when it's an exceptional project that doesn't use a message queue, rather than the other way around.