Some people wanted us to build a big powerful SMP system. Some people wanted us to build a silent machine that would be good for audio. So we did both.
Let's just call 2005 the year of power management. Processor vendors made a big deal out of whitepapers about saving watts, and we heard a lot about power management at LinuxWorld Conference and Expo in February.
Did the industry start caring about global warming? Do IT CEOs want to eat swordfish more often, so they have to reduce the mercury emissions of power plants? Not quite. Today's server systems are packing more and hotter processors closer together, and customers' air-conditioning systems aren't ready for the strain. NASA had to install water cooling for its 10,240-processor Columbia cluster, as we showed in our January issue.
Every watt-hour you can save is heat that the customer doesn't have to deal with—3.6kJ, or 3.4BTU to be precise. With data centers full of blade servers, and 1U systems sporting as many as four processors, all that heat really adds up.
The Linux desktop greedily devours the scraps from the multibillion dollar Linux server market, and power consumption matters to us on the desktop too. Fans are loud. If you have better power management on your processors, they produce less heat, and you can run fewer fans or run the fans you do have more quietly. We took a different approach to fans, as you'll see later on.
Finally, of course, power matters on the laptop and on portable devices because of battery life. We'll leave the specifics of tweaking for maximum off-AC time to future articles.
We like Tyan motherboards, and companies that build custom Linux systems do too. The four-Opteron Tyan Thunder K8QS Pro came out just a little too late to make it into last year's Ultimate Linux Box. It's based on an AMD 8000 series chipset. When we say “chipset”, we mean a slightly different combination of hardware from an Intel-based system, though. The AMD64 way is to have an onboard memory controller per processor, give each processor its own bank of memory and link them with HyperTransport. Your AMD64 “SMP” box is really a mini-NUMA, and the “chipset” doesn't include the memory controller.
Last year, we used a Celestica A8440 bare-bones rackmount system as the basis for the Ultimate Linux box. Although starting with pre-integrated chassis and power supplies can be a great time saver, we realized that last year's box was on the loud side. This year, going back to our usual plan lets us pick everything else just the way we want it.
The K8QS Pro has two PCI-X busses, A and B. B is dedicated to two 133MHz-capable PCI-X slots, and A offers two PCI-X slots maxing out at 66MHz and one regular PCI slot. Onboard networking is two Broadcom BCM5704C Gigabit Ethernet interfaces, also on bus A.
There are all the regular PC ports, of which we're using the USB. SCSI and serial ATA are options, which you might want to keep in mind if you're planning to move this board into a more conventional server role when you're building your next Ultimate Linux Box.
Into this mighty board we plugged four of the best of the Opteron processors available at the time—the 846 HE, clocked at 2.0GHz and offering 1MB of L2 cache. See the sidebar for what became available while we were testing the system. We maxed out the system's main memory at 32GB.
Unfortunately for case shoppers, this board is SSI MEB size—13"x16" or 330.2x406.4mm. Not a problem for us because we're using a custom case this year, but the size does limit your case options.
When we're picking out a case for any custom-built system, Ultimate or otherwise, we usually get one that's quite a bit larger than what a big vendor would use for a comparable system. Smaller cases require less material and they're cheaper for vendors to ship, but since we like to tweak things, we get a case with more room to add devices and more room to work inside.
In order to have a completely silent system, you need to move storage outside the box. Options for doing this have changed a lot since the days when you had a choice between NFS and external SCSI enclosures connected by a 3-meter cable.
Today, you can make your drives go away using USB, FireWire, SCSI of course, Fibre Channel or the new ATA over Ethernet, which we covered in the June 2005 issue. A separate storage enclosure is no longer only an enterprise server-room thing.
Another option is simply to boot over the network and mount your storage via NFS. Since Penguin works with enterprise server-room hardware, and Fibre Channel does deliver impressive benchmark results, we went with it; an nStor 4320F Fibre Channel RAID enclosure, with Hitachi 18GB drives for the OS and larger Seagate drives for more storage.
Because we wanted the system to be self-contained and not depend on another server to boot, we installed a Sandisk 256MB CompactFlash card to boot from. This device looks exactly like another ATA drive to the system, so any PC motherboard will boot from it.
We considered using a USB thumbdrive, but that would have required some initrd drive juggling and GRUB wizardry. There are advantages to being able to pull your boot device out of the system and store it separately, but we didn't anticipate shipping the system through airports with drives loaded with encrypted confidential data.
If you plan to leave your silent Linux system on your network, you'll be a little more flexible in booting, and you can set up PXE booting. But if you want to take your Ultimate Linux Box over to a friend's house to play some music, you'll want to be able to boot independently. The Penguin crew plans to take this system to LinuxWorld Conference and Expo, and when you're wrangling hardware for a tradeshow one fewer thing to set up is good.
If you do build and install a silent Linux box, you'll probably end up doing a mix of both: NFS for user home directories, the company /usr/local/bin/ and other items that need to be in sync but aren't performance-critical. You can save your machine's own filesystems for big working files, like all the audio data you'll get from this system's high-end sound hardware.
Finally, to take even the keyboard clicking out of the silent system, Penguin founder Sam Ockman suggested a TouchStream LP keyboard, which works like a touchpad and requires no moving parts. It's also a pointing device and lets you map gestures to interface actions.
For the first time, we put professional audio hardware into the Ultimate Linux Box. What better place for a silent machine than the recording studio?
The RME Hammerfall HDSP9652 card we chose for this system is capable of up to 52 channels, and we matched it with an external box called the Multiface that brings out 8 1/4" jacks, as well as optical, coax and MIDI.
This card is as close as you can get to a “studio in a box”, because it's built around an internal mixer and allows you to route signals around inside the card with low latencies and low load on the CPU. Other features include the ability to “punch in” and “punch out” like a conventional tape deck.
Best of all, RME has been supporting the Advanced Linux Sound Architecture (ALSA) Project since 2000, so Linux users aren't second-class citizens. RME's site says, “ALSA support for the Hammerfall breaks the annoying chicken/egg principle—no professional hardware/driver, no professional software.”
Peter Todd covered the necessary tools for working with the Hammerfall HDSP cards in our October 2003 issue.
For video, we used a relatively low-end card (see the on-line Resources). We'd really like to start putting interesting and innovative video on Ultimate Linux Boxes, but there are still some issues with the drivers (see sidebar).
So how do we keep this thing cool? First of all, it's important not to start tweaking with hardware combinations unless you know how to measure the effects that your changes have on the system's temperature. Don't change anything unless you know how to measure the effect of the change.
The good news is that the processor and motherboard vendors thoughtfully give us temperature sensors right on the key parts. And we can keep track of them using an all-important tool, lm_sensors.
We didn't have to measure drive temperature because we moved the drives to a separate enclosure, but smartmontools (see Resources) gives you an easy way to do that.
We ordered up some parts from Zalman, which offers a beautiful set of water-cooling hardware. The most visible part is the Reserator 1, a combined water reservoir and radiator that stands half a meter tall and holds 2.5 liters of water. Besides the Reserator, we also ordered one CPU waterblock per processor and matching tubing.
Thermal estimates showed that we wouldn't need a full Reserator per processor, so we used one Reserator per two processors and one for the power supply.
The Reserator comes with a 5W pump, which would break our beautiful silence, so it was time to convert it to operate purely by convection. In its stock configuration, the Reserator's inlet and outlet are close to each other, so we installed a tube inside each Reserator, running from the hot inlet to near the top.
What's that in your cubicle, Justin? We tested convective cooling with a scratch system and lm_sensors.
Did it work? The processor temperature climbed to about 50° C, then the tubes leading up from the processors to the Reserators warmed enough to start the convection. Temperature fell to 47° or 48° C in normal use, and running full-out, the system holds out below 50° C.
Cooling the power supply was a little harder. Zalman's beefiest fanless power supply is only 400W, and a big four-way board needs more. We decided to use the PC Power and Cooling Turbo-Cool 510 ATX.
We decided not to design and build a power supply for the project, since it's important to apply power to components in the right order, and we know PC Power and Cooling solved that problem for us. The cooling problem remained.
Enter the magic of metalworking. Phil brought the problem to a machine shop called Global Precision, and we had them do three pieces of work. They machined down the original fins of the power supply's heat sinks to create flat areas for attaching waterblocks. They made the waterblocks themselves—using blue anodized aluminum to match the Zalman parts. And they made two custom Y-connectors to split the water flow between the two heat sinks.
We removed the fan control board from the power supply. We didn't need it any more.
Cases capable of accommodating and doing justice to Ultimate Linux Boxes are rare. This year, only one alternative would work: going full custom. This year's case has acrylic windows to show off the cooling system, integrated supports for the three Reserators and a mounting place for the RME Multiface.
Look, everybody, no leaks! Justin sets up for the cover photo shoot (photo: Don Cameron).
Difficult as it might be for us to believe right now, many real-world systems don't need both 52-channel audio and Fibre Channel. But unusual combinations of hardware are what enable creative projects, and we're happy that Linux stays out of our way and lets us hook up what we want.
When you start with what's possible and take out what you don't need, you'll be confident that you can build a machine for your needs. We hope that whatever class of system you decide to build, you'll get some ideas out of this year's Ultimate Linux Box.
Resources for this article: /article/8330.