The Ultimate Linux Server

We defined specific parameters that would describe our Ultimate Linux Server before we began our search. To put it simply, the server had to be built for reliability, high availability and provide the best bang for the buck. Only one vendor provided us with a server that met our requirements. Although this made it easy to choose the Aberdeen Stonehaven A261T dual-core dual-processor rackmount server as our Ultimate Linux Server, one shouldn't get the idea that the Stonehaven doesn't deserve its place at the top based on its own merits. This server meets our entry criteria and then some.

The Aberdeen Stonehaven A261T rackmount server has two dual-core AMD Opteron processors for a total of four effective CPUs, and it comes with six 250GB hard drives configured for RAID 5. Aberdeen can provide up to 12TB of storage if you need it. The server comes with redundant power supplies for increased availability. Should one fail, you can hot swap a new one in its place. You also can hot swap the SATA drives.

The A261T is based on the Tyan S2882-D Thunder K8SD Pro motherboard. Tyan has long been a reliable provider of server motherboards, and this particular model is a terrific foundation upon which to build. The server includes an Areca SATA RAID controller. The default Linux software is Red Hat Enterprise Linux Server 4.0, 64-bit (x86_64), which is included with the server along with a handful of driver disks, manuals and other minor goodies.

Here is the official list of the box's features that we tested:

We ran a series of benchmarks on the A261T. Most of them produced meaningless results, because the A261T was too fast for our benchmarks to stress the hardware adequately. We finally produced our own benchmark based on the siege program, which simulates a number of simultaneous Web clients hammering the server for Web pages. It wouldn't do to serve up static pages for this benchmark, because that would create no significant stress on the server. So we created a Web site based on Apache, PHP and MySQL that includes IFrames, JavaScript and records information about each visitor, so that each page access produces write operations on MySQL as well as read operations.

This turned out to be too much for the server at first. In fact, our dinky workstation outperformed the server by a wide margin. Naturally, this made no sense, so we went on a troubleshooting spree. The answer turned out to be a poorly configured Apache file (httpd.conf). We suspect that the fault lies primarily with the GUI httpd configuration tool in Red Hat Enterprise Linux, but we haven't had time to back out all the changes and try the process from scratch in order to identify where things went wrong. Regardless, this is an issue with Red Hat EL 4.0, not with the server hardware. Any administrators worth their salt will want to tweak the http.conf one way or another to adapt it to their applications, whether they do so manually or via any one of several Web-based or GUI tools.

We didn't find this problem easily, but that's thanks mostly to Murphy's Law, which includes the axiom that one never looks in the right place the first time. At first, we noticed that MySQL was using up an unusual amount of CPU power. Was the outdated MySQL that comes with Red Hat Enterprise Linux 4.0 causing the problem? No. We replaced the 4.x version of MySQL with a 5.x version, and it made no difference. We ran up2date and chose to replace the existing Linux kernel with the latest version. This turned out to be a bigger task than expected. The Red Hat kernel doesn't include support for the Areca RAID controller. Fortunately, Aberdeen provides a CD-ROM with driver code and instructions for creating a module for any new Red Hat kernel and instructions on including this module in the initial RAM disk. Once we found those instructions, it was just a matter of minutes before we had the new kernel up and running. It didn't matter. The new kernel and driver didn't improve performance.

That's when we finally looked at the Apache httpd.conf file. On a hunch, we removed a duplicate Alias line created by Red Hat's GUI tool, and we tweaked the pre-fork parameters of Apache. Here is what we ended up using:

<IfModule prefork.c>
StartServers 20
MinSpareServers 5
MaxSpareServers 20
MaxClients 256
MaxRequestsPerChild 0
</IfModule>

We restarted Apache and MySQL, re-initialized the database and Web site and then nothing seemed to be able to slow this puppy down. The siege benchmark defaults to simulating 15 simultaneous clients. Don't be deceived by this apparently tiny number of clients. The siege benchmark doesn't merely send 15 clients to the Web site, it hammers away at the Web site continually, unlike how 15 real users would. Real people read a page and move on. The siege benchmark doesn't bother reading anything. It just keeps hammering away.

Nevertheless, the A261T laughed at our 15 simultaneous users. So, we kept boosting the number until we hit 500 simultaneous users, which produced enough load to make the server recognize we were there. Table 1 shows the final results of the siege test with 500 simultaneous users.

As you can see, the response time was usually less than one second, which is remarkably fast for the kind of pages we were serving. The server had no problem delivering more than 300 transactions per second with a concurrency of 218 requests.

We had to go back to our workstation and try the same test, given that the workstation previously left the A261T server in the dust at 15 concurrent users. We changed the workstation configuration to match the server, and the workstation couldn't even finish the test at 500 concurrent users. In fact, the only way we could get it to finish was to reduce the number of concurrent users to 100 maximum, and even then it failed one transaction. We produced the numbers in Table 2 on a system with an AMD 4400+ Athlon x2 processor with 4GB of DDR400 RAM and a 300GB SATA 3.0Gb/s drive (no RAID).

This story wouldn't be complete without mentioning our personal blunders. The Aberdeen server made an extremely loud whining noise when we first set it up. We tolerated this noise for what seemed like an eternity (more like two weeks of playing around with the server and running the original benchmarks, which we eventually had to replace). When we'd had enough, we opened up the case to listen to specific areas and feel around for vibrations in order to locate the source of the noise. A slip of the wrist, and suddenly there was a piece of thumb caught in a CPU fan blade. (Never let it be said that we don't put a bit of ourselves into everything we do.) Now we had not only a loud whine, but also a fan that made a nearly unbearable racket.

It turns out the whine was just the server's way of letting us know one of the redundant power supplies wasn't working. Why wasn't it working? Because we didn't have the power cord pushed in all the way. It turned out we could have saved ourselves two weeks of agony and a band-aid if we'd simply pushed in the power cords more carefully. As for the fan, it worked, if noisily. But we replaced it with one of our own while we waited for Aberdeen to send another. The server wasn't whisper quiet by any means once we had it in normal operation, but it sounded like virtually any other rackmount server that has to push a lot of air in order to stay cool.

The retail direct price for this particular configuration is $4,999 US. Given the design for reliability and its excellent performance and scalability, we consider the A261T to deliver an awful lot of bang for this buck. This model is very well deserving of our title of Ultimate Linux Server, and we recommend the Aberdeen line of servers without reservation.