At the Forge

The past few months, I've covered a number of different non-relational (NoSQL) databases. Such databases are becoming increasingly popular, because they offer both easier (and sometimes greater) speed and scalability than relational databases typically can provide. In most cases, they also are “schemaless”, meaning you don't need to predefine or declare the names, types and sizes of the data you are storing. This also means you can store persistent information with the ease and flexibility of a hash table.

I'm still skeptical that these non-relational databases always should be used in place of their relational counterparts. Relational databases have many years of thought, development and debugging behind them. But, relational databases are designed for reliability and for arbitrary combinations of data. NoSQL databases, by contrast, are designed for speed and scalability, without “joins” and other items that are a central pillar of relational queries.

Thus, I've come to believe that relational databases still have an important role to play in the computer world, and even in the world of high-powered Web applications. However, just as the introduction of built-in strings, arrays, hash tables and other sophisticated data structures have made life easier for countless programmers, I feel that non-relational databases have an important role to play, offering developers a new mix of interesting and useful ways to store and retrieve data.

To date, I have explored several non-relational systems in this column. CouchDB and MongoDB are both “document” databases, meaning they basically allow you to store collections of name-value pairs (hashes, if you like) and then retrieve elements from those collections using various types of queries. CouchDB and MongoDB are quite different in how they store and retrieve data, and they also approach replication differently.

Both CouchDB and MongoDB are closer in style and spirit to one another than to the system I covered last month, Redis—a key-value store that's extremely fast but limits you to querying on a particular key, and with a limited set of data types. Plus, Redis assumes you have a single server. Although you can replicate to a secondary server, there is no partitioning of the data or the load among more than one node.

Cassandra is a little like all of these, and yet it's quite different from any of them. Cassandra stores data in what can be considered a multilevel (or multidimensional) hash table. You can retrieve information according to the keys, making it like a key-value store, like Redis or Memcached. But, Cassandra allows you to ask for a range of keys, giving it a bit of extra flexibility. Moreover, the multidimensional nature of Cassandra, its use of “super columns” to store multiple items of a similar type and its storage of name-value pairs at the bottom level provide a fair amount of flexibility.

Cassandra really shines when it comes to many aspects of scalability. You can add nodes, and Cassandra integrates them into the storage system seamlessly. Nodes can die or be removed, and the system handles that appropriately. All nodes eventually contain all data, meaning even if you kill off all but one of the nodes in a Cassandra storage cluster, the system should continue to run seamlessly. Because writes are distributed across the different nodes, it takes a very short time to write new data to Cassandra.

It's clear that Cassandra has resonated with a large number of developers. The project started at Facebook, in order to solve the problem of searching through users' inboxes. Facebook donated the code to the Apache Project, which has since promoted it and made it a first-class project. Facebook no longer participates in the open-source version of Cassandra, but apparently Facebook still uses it on its systems. Meanwhile, companies including Rackspace, Twitter and Digg all have become active and prominent Cassandra users, contributing code and contributing to the general sense of momentum that Cassandra offers.

Perhaps the two biggest hurdles I've had to overcome in working with Cassandra are the unusual terminology and the configuration and administration that are necessary. The terminology is difficult in part because it uses existing terms (“column” and “row”, for example) in ways that differ from what I'm used to with relational databases. It's not hard, but does take some getting used to. (Although the developers might have done everyone a favor by avoiding such terms as “column families” and “super columns”.) The configuration aspects aren't terribly onerous, but perhaps point to how spoiled people have gotten when working with non-relational databases. The fact that I have to name my keyspaces and column families in a configuration file, and then restart Cassandra so that their definition will take effect, seems like a throwback to older, more rigid systems. However, relational databases force us to define our tables, columns and data types before we can use them, and it never seemed like a terrible burden. And, it seems that part of the secret of Cassandra's speed and reliability is the fact that its data structures are rigidly defined.

This month, I take an initial look at getting Cassandra up and running and explain how to store and retrieve data inside a simple Cassandra instance.

The home page for Cassandra is cassandra.apache.org. From there, you can download Cassandra and install it on your computer. Because Cassandra is written in Java, there is only one distribution binary, which should work on any computer with a current JVM.

On my computer running Ubuntu, I first installed the latest Java JDK with:

apt-get install openjdk-6-jdk

Following this, I could have downloaded the latest Cassandra version and installed it. But instead, I decided to use apt-get to retrieve the latest version and to ensure that I will receive updates in the future. In order to do this, I first needed to add the appropriate GPG keys to my keychain, as per the instructions on the Cassandra Wiki:

gpg --keyserver wwwkeys.eu.pgp.net --recv-keys F758CE318D77295D
gpg --export --armor F758CE318D77295D | sudo apt-key add -

Following that, I added these two lines to /etc/apt/sources.list:

deb http://www.apache.org/dist/cassandra/debian unstable main
deb-src http://www.apache.org/dist/cassandra/debian unstable main

Next, I ran apt-get update to retrieve the latest version information for all packages, and then I ran apt-get install cassandra to install it on the server. About a minute later, Cassandra was installed and ready to run on my machine.

I started it up with:

/etc/init.d/cassandra start

Sure enough, a quick peek at ps showed me that Cassandra indeed was running.

There are numerous interfaces to Cassandra from a variety of programming languages. However, the easiest way to connect to Cassandra often is via its built-in command-line interface (CLI), which comes with the program. Simply enter cassandra-cli in your shell, and you'll see a prompt that looks like this:

Welcome to cassandra CLI.

Type 'help' or '?' for help. Type 'quit' or 'exit' to quit.
cassandra>

Your first task should be to connect to your local Cassandra server:

cassandra> connect localhost/9160
Connected to: "Test Cluster" on localhost/9160

In case you forgot what was just printed, you can get the current cluster name with:

cassandra> show cluster name
Test Cluster

You also can get a list of keyspaces in this cluster:

cassandra> show keyspaces
Keyspace1
system

The system keyspace, as you can imagine, is used for Cassandra system tasks. It can be fun and interesting to explore, but you don't want to mess with it unless you really know what you're doing.

What if you want to create a new keyspace? Well, that's where you'll need to go in and change the system's configuration and restart Cassandra. The configuration file you need to modify is called storage-conf.xml. After I installed Cassandra on my Ubuntu system, it was placed in /etc/cassandra/storage-conf.xml. (The filename always will be storage-conf.xml, but the location might differ on your machine, depending on how you installed it.) You can see the contents of this configuration file from the Cassandra CLI, with the command:

cassandra> show config file

However, this command shows only the contents of the file, not its location, so you might have to poke around a bit to find it.

To add a new keyspace to your Cassandra cluster, first you must think about what you want to store and then how you can represent that in Cassandra. As an example, let's store a list of users. You don't need to think beyond that right now; all you need to define is the name of your column family. Individual columns and values can and will be defined on the fly.

To do this, define a new keyspace and one new column family. Each column family is analogous to a table in a relational database; it contains zero or more columns. Each column, in turn, is a name-value pair. Thus, by defining your keyspace as follows, you're basically saying you want to store information about users:

<Keyspace Name="People">
<ColumnFamily Name="Users" CompareWith="BytesType"/>
</Keyspace>
</Keyspaces>

Like a relational database, you'll be able to store many fields of information about these users. Unlike a relational database, you don't need to define them from the start. Also unlike a relational database, you'll be able to retrieve information about users only via the key you use for this column family. So, if you use e-mail addresses as keys into the “Users” column family, you'll need an address to do something; having the person's first and last name will not do you much good.

Cassandra stores information as a set of bytes; there are no internal types. However, you can (and should) indicate to Cassandra how the data should be sorted. Specifying a “comparator” allows you to simulate the storage of different types. More important, it determines the order in which you will receive results. That's because there is no ORDER BY equivalent in Cassandra when you retrieve data; you need to decide on an order and specify it in the configuration file. Somewhat surprisingly, the ordering is done when the data is written, not when it is read. In the case of the example “Users” column family, you'll just retrieve them in byte order.

If you put the above <Keyspace> section inside the <Keyspaces> tag in your storage-conf.xml file and restart Cassandra, you'll find that it fails to start up. (The error logs are in /var/log/cassandra, at least in my Ubuntu installation.) That's because there are three other definitions you need to include: ReplicaPlacementStrategy, ReplicationFactor and EndPointSnitch. None of these definitions will concern you when you have a single Cassandra node, so I suggest simply copying them from the included Keyspace1 keyspace. In the end, this part of your keyspace definition will look like this:

<Keyspace Name="People">
<ColumnFamily Name="Users" CompareWith="BytesType"/>

<ReplicaPlacementStrategy>org.apache.cassandra.locator.
↪RackUnawareStrategy</ReplicaPlacementStrategy>
<ReplicationFactor>1</ReplicationFactor>
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch
↪</EndPointSnitch>
</Keyspace>

Restart Cassandra, and reconnect via the CLI. Then, type:

cassandra> show keyspaces

Your new keyspace, “People”, now should appear in the list:

cassandra> show keyspaces
Keyspace1
system
People

You can ask for a description of your keyspace:

cassandra> describe keyspace People
People.Users
Column Family Type: Standard
Columns Sorted By: org.apache.cassandra.db.marshal.BytesType@1b22920

Column Family Type: Standard
Column Sorted By: org.apache.cassandra.db.marshal.BytesType
flush period: null minutes
------

You now can see that your People keyspace contains a single “Users” column family. With this in place, you can start to set and retrieve data:

cassandra> get People.Users['1']
Returned 0 results.

cassandra> set People.Users['1']['email'] = 'reuven@lerner.co.il'
cassandra> set People.Users['1']['first_name'] = 'Reuven'
cassandra> set People.Users['1']['last_name'] = 'Lerner'

In Cassandra-ese, you would say that you now have set three column values ('email', 'first_name' and 'last_name'), for one key ('1') under a single column family (“Users”), within a single Keyspace (“People”). If you're used to working with a language like Ruby or Python, you might feel a bit underwhelmed—after all, it looks like you just set a multilevel hash. But that makes sense, given that Cassandra is a super-version of a key-value store, right?

Now, let's try to retrieve the data. You can do that with the key:

cassandra> get People.Users['1']
=> (column=6c6173745f6e616d65, value=Lerner, 
 ↪timestamp=1279024194314000)
=> (column=66697273745f6e616d65, value=Reuven, 
 ↪timestamp=1279024183326000)
=> (column=656d61696c, value=reuven@lerner.co.il,
timestamp=1279024170585000)
Returned 3 results.

Notice how each column has its own unique ID and that the data was stored with a timestamp. Such timestamps are crucial when you are running multiple Cassandra nodes, and they update one another without your knowledge to achieve complete consistency.

You can add additional information too:

cassandra> set People.Users['2']['first_name'] = 'Atara'
cassandra> set People.Users['2']['last_name'] = 'Lerner-Friedman'
cassandra> set People.Users['2']['school'] = 'Yachad'

cassandra> set People.Users['3']['first_name'] = 'Shikma'
cassandra> set People.Users['3']['last_name'] = 'Lerner-Friedman'
cassandra> set People.Users['3']['school'] = 'Yachad'

Now you have information about three users, and as you can see, the columns that you used within the “Users” column family were not determined by the configuration file and can be added on the spot. Moreover, there is no rule saying that you must set a value for the “email” column; such enforcement doesn't exist in Cassandra. But what is perhaps most amazing to relational database veterans is that there isn't any way to retrieve all the values that have a last_name of 'Lerner-Friedman' or a school named 'Yachad'. Everything is based on the key (which I have set to an integer in this case); you can drill down, but not across, as it were.

You can ask Cassandra how many columns were set for a given key, but you won't know what columns those were:

cassandra> count People.Users['1']
3 columns
cassandra> count People.Users['2']
3 columns

However, if you're trying to store information about many users, and those users are going to be updating their information on a regular basis, Cassandra can be quite helpful.

Now that you've got the hang of columns, I'll mention a particularly interesting part of the Cassandra data model. Instead of defining columns, you instead can define “super columns”. Each super column is just like a regular column, except it can contain multiple columns within it (rather than a name-value pair). In order to define a super column, set the ColumnType attribute in the storage-conf.xml file to “Super”. For example:

<ColumnFamily Name="Users" CompareWith="BytesType" 
 ↪ColumnType="Super" />

Note that if you restart Cassandra with this changed definition, and then try to retrieve People.Users['1'], you'll probably get an error. That's because you effectively have changed the schema without changing the data, which always is a bad idea. Now you can store and retrieve finer-grained information:

cassandra> set People.Users['1']['address']['city'] = 'Modiin'

cassandra> get People.Users['1']['address']['city']
=> (column=63697479, value=Modiin, timestamp=1279026442675000)

Cassandra provides a non-relational storage and retrieval mechanism (NoSQL database) that features tremendous scalability, speed and flexibility. The inclusion of super columns (and super-column families, which I didn't discuss here) gives you just enough flexibility to store a great deal of information about many users. So long as you never have to search on anything other than the primary key or join information from different users at the database level, Cassandra is a good choice.

That said, Cassandra is significantly harder to understand and administer than other non-relational databases. I think the investment of time and effort are worth it, but you shouldn't expect to be able to work with Cassandra as quickly and easily as with, say, CouchDB or MongoDB. The flip side of this issue is that administration allows you to fine-tune a number of aspects of Cassandra's networking and consistency until you reach a level with which you're comfortable.

Next month, I'll continue exploring and discussing Cassandra, looking at ways to connect multiple Cassandra boxes to a cluster—and what happens when you do so.