Focusing on maximum crunching power and PC hardware whendeveloping the Ultimate Linux Box (ULB).
Five years ago, I wrote an article for Linux Journal that developed a recipe for an elegant and economical Linux box. The article became one of the most popular in LJ's history, so the editors have invited me back for a second round.
This time around LJ recruited Rick Moen, author of some well-known FAQs on modems and other hardware topics, to assist. Daryll Strauss, the man behind the famous Linux renderfarm used in the movie Titanic, contributed sage advice from his background in graphics and extreme data crunching. Also, instead of going for economy we're going to go for maximum crunching power. We're going to ask how to get the absolute highest performance out of hardware we can live with.
Hardware you can live with means a machine that is stable, easy to troubleshoot and inexpensive to maintain. It should be small and low-maintenance enough to live beside your desk, not some liquid-cooled monstrosity. It should be, in short, a PC—a gold-plated hot rod of a PC but a PC nevertheless. Other important aspects of livability are the levels of emitted acoustic noise and heat; we'll be minimizing both.
We'll stick with PC hardware. Alphas are fast and have that wonderful, sexy 64-bit architecture, but the line seems all too likely to be killed in favor of the Itanium before long. Considered in isolation, I like the PowerPC chip a lot better than any x86 architecture. But PC hardware has all the advantages of being the biggest market; it's the easiest to get serviced and least expensive to upgrade.
The Ultimate Linux Box that we showcase will, of course, fall behind the leading edge within months (or even by press time). But walking through the process of developing the ULB will teach you things about system design and troubleshooting that you can continue to apply long after the hardware in this article has become obsolete.
For typical job loads under Linux, the processor type is nearly a red herring—it's far more important to specify a capable system bus and disk I/O subsystem. If you don't believe this, you may find it enlightening to keep top(1) running for a while as you use your machine. Notice how seldom the CPU idle percentage drops below 90%.
If you're building a ULB, go ahead and buy the fastest available processor. Once you've gotten past that gearhead desire for big numbers, pay careful attention to bus speeds and your disk subsystem because that's where you'll achieve serious performance wins. The gap between processor speed and I/O subsystem throughput has only widened in the last five years, so this is even better advice than it was in 1996.
Everybody's Doing Dual Athlons
How does all this translate into a recipe in 2001? Get a PCI-bus machine, not a hybrid PCI/ISA design; you sacrifice about 10% of peak performance with those. Get the fastest available front-side (processor-to-memory) bus (in August 2001, the maximum is 266MHz). Get a high-speed SCSI controller and the fastest SCSI disks you can get your hands on.
The case for SCSI is a little less obvious but still compelling. For starters, SCSI is still at least 10%-15% faster than IDE/ATAPI running flat out. Because it's perceived as a professional choice, SCSI peripherals are generally better engineered than IDE/ATAPI equivalents, and new high-performing drive technologies tend to become available in SCSI first. You'll pay a few dollars more, but the cost is well repaid in increased throughput and reliability. Rick Moen comments:
They call me a SCSI bigot. So be it—but my hardware keeps being future-proof: I don't have to run bizarre emulation layers to address CDRs, I never run low on IRQs or resort to IRQ-sharing, all my hard drives have hardware-level hot-fix, all my hard disk/CD/tape/etc., devices support a stable standard rather than this month's cheap extension kludge, and I don't have to worry about adverse interactions at the hardware or driver levels from mixing ATA and SCSI.
Neither Daryll nor I will have IDE in any machine we build either.
To pick the fastest disks, pay close attention to average seek and latency time. The former is an average time required to seek to any track; the latter is the maximum time required for any sector on a track to come under the heads.
Of these, average seek time is much more important. When you're running Linux, a one millisecond faster seek time can make a substantial difference in system throughput. The manufacturers themselves avoid running up seek time on larger-capacity drives by stacking platters vertically rather than increasing platter size. Thus, seek time, which is proportional to the platter radius and head-motion speed, tends to be constant across different capacities in the same product line. This is good because it means you don't have to worry about a capacity vs. speed trade-off.
I always build with two disks: one system disk and one home disk. There are two good reasons to do this that have nothing to do with the extra capacity. One is the performance advantage of being able to interleave commands to different physical spindles. The other is I am quite a bit less likely to lose two disks at once than I am to trash a single one.
Let's suppose you have a fatal disk crash. If you have only one disk, good-bye Charlie. If you have two, maybe the crashed one was your system disk, in which case you can buy another and do a new Linux installation, knowing your personal files are safe. Or maybe it was your home disk; in that case, you can buy another home disk and restore it from backups (you did keep backups, right?).
Debian, Installing on Hardware Your Distribution Doesn't Support and the ReiserFS Two-Step
Max out your memory. Lots of free memory will improve your virtual-memory performance. Fortunately, with RAM as cheap as it is now, a gigabyte or three is unlikely to bust your budget even if you're economizing.
You'll need a CD-ROM or DVD-ROM drive (you'll almost certainly be installing your Linux from it). You have a SCSI system, so get a SCSI CD-ROM. That's pretty much the end of spec, as there are only a few models of SCSI DVD-ROM, and SCSI CD-ROMs are a generic item.
We'll want a good, high-volume backup device, too. Large disks are so cheap that backing up your home directory to another disk seems an attractive alternative, but it's still good to be able to make backups that you can separate from your system and store off-site, in case of disaster. We'll go with a DDS tape drive. Even if you're building on the cheap, the less expensive CD-ROM burners aren't a good idea for mass backup. The problem is the per-megabyte cost of the media, which you can't reuse. Rick adds: “Tape is also faster, more rugged both in storage and in the process of recording (jostling a DAT drive doesn't destroy the ongoing backup), doesn't require gobs of scratch space for assembling image files and is way, way, easier to automate.”
Speaking of faster, one of the things you want most in a tape drive is transfer speed. This is a good reason to go with the newer DDS4 tape drives, which have speed that is typically half of the older DDS3 drives.
An increasingly critical aspect of machine design is handling the waste heat and acoustic noise of operation. Cooling is centrally important if you want your ULB to last because thermal stress from waste heat is almost certainly what will kill it. On the other hand, cooling makes acoustic noise, which human beings don't tolerate well. It's fair to say we've already reached the point at which the thermal load vs. cooling-noise trade-off is the effective limiting factor in the performance of personal machines.
So how do we manage this trade-off for a personal, desktop or desk-side machine? Being willing to pay a price premium for cool-running and low-noise parts can help a lot. Even clueful system integrators can't afford to do this because they're under constant competitive pressure to cut costs by using generic components. But, we aren't economizing here; we get to do it right.
Now that we've laid out the principles, it's time to do the practice—specify and build a machine.
In July 2001, the clear standout choice for a ULB motherboard is the Tyan Thunder K7, model S2462 (see the Sidebar titled “AMD, SMP, AGP and LEDs: the Tyan Thunder K7 S2462”).
AMD, SMP, AGP and LEDs: the Tyan Thunder K7 S2462
There are good and bad consequences of having your peripherals onboard. The good ones are that the board has fewer points of failure and will throw less heat. The downside is that integration could make fault recovery more difficult. You want to minimize the chance that a failure in one onboard component will require an immediate motherboard swap. On the S2462, all the onboard peripherals can be jumpered out or disabled from the BIOS setup screens.
Internal expansion space isn't very important anymore because two-drive bays will hold more disk than you'll ever need. External bays are more important; you want one CD-ROM, one tape, one floppy and perhaps a DVD drive. That's one exposed floppy bay, three exposed half-height 5.25" bays and two internal bays.
There are three other important things you want from a case: good airflow design, component accessibility and noise attenuation, in that order. Finally, you may want your case to look neat. Good airflow design is actually the best reason to buy a large case. You want plenty of room for cool air to flow around the heat-generating electronics.
Tyan's site lists cases that have been qualified with the S2462, so I shopped around for a full tower on that list. Antec's Performance Series offers a number of cases that Tyan qualifies, and the swing-out side panel and quick-release drive bays featured on all of them appealed to me. When my design evolved to include a DVD player and the front-panel controls for a sound card, I went with the SX1200, the full-tower version with seven exposed bays.
For the power supply, the three of us agreed on a vendor: PC Power & Cooling. PCP & C has a reputation for making good units and, as a bonus, quiet ones. PCP & C justified our confidence when they told me of their brand-new 450A4 unit, specifically designed for use with the S2462. And at 44dBA, the A4 counts as pretty quiet.
In May, Tom's Hardware compared 46 CPU coolers. The clear standout is the Silverado from Noise Control, Inc., rated best in cooling performance at 30°C and second-best in noise emission, only 37dBA. Typical coolers emit about 50dBA. The Silverado's only real drawback is that it's large—80mm long, 56mm wide, 113mm high—so you need to be careful about case clearances.
We can avoid having our case fans add more than a bare minimum to the machine's decibel output by specifying cooling fans that have ball bearings rather than the cheaper and more common sleeve bearings. This will cut machine noise by an appreciable degree, especially the annoying, whining high-frequency component, which is mostly bearing noise.
PC Power & Cooling makes 20dBA Silencer 80mm ball-bearing case fans. Specify the three-pin connectors to plug into the motherboard, not the four-pin connectors meant to be plugged into the power supply.
We're going to be specifying fast-wide LVD drives, the cutting edge in SCSI devices. Within that class, the important statistics are seek time, rotational latency, capacity, heat dissipation and noise output. Mean time between failure is long enough on the leading brands that you're quite unlikely to see one before your system is years obsolete.
A StorageReview.com search confirmed anecdotal evidence from Rick Moen. He likes IBM's current product line, the UltraStar. With a 4.2ms seek time, they edge ahead of competition from Seagate, Quantum and Fujitsu. Rick believes they run relatively cool, too, and we hear they smoked the competition in some comparative trials run by Evi Nemeth at the CAIDA Project. So we'll add two IBM UltraStar 36Z15 drives to the parts list.
We also want to be able to read (and write) CD-ROMs. Again, StorageReview.com confirms Rick's anecdotal report, tapping the 32-speed Plextor PX W1210TS as the best-of-breed among SCSI CD-RW drives.
Rick observes:
CD-R/CD-RW drives by their nature have head assemblies much more massive than those of ordinary read-only CD drives. Why? Because they mount burn lasers. Much greater mass means much greater inertia and much faster mechanical wear, and the considerable heat generated during burn cycles also takes its toll. Accordingly, the MTBF times for CD-R/CD-RW drives are markedly shorter than for regular CD drives. One should not use CD-R/CD-RW drives for mundane read operations, but rather only for CD-burning. Accordingly, if you really have the need for a CD-R or CD-RW drive, you also need a second, read-only drive for everyday CD-reading.
Daryll Strauss chimes in with: “Buy a DVD-ROM rather than an ordinary CD-ROM. Typically the transfer rates are just as good, if not better, because the base DVD rotational speed is higher to begin with.”
A DVD is a must-have for another reason; any true dream system for a Linux hacker must include the ability to violate the anti-fair-use clauses of the Digital Millennium Copyright Act by playing DVDs, even if (like me) the hacker is basically uninterested in DVDs per se. It's ethically imperative.
Presently, only two models of SCSI DVD-ROM are available: the 304S/305S by Pioneer and the SD-M1201B by Toshiba. The Toshiba is 5X as a DVD drive and 32X as a CD-ROM drive; the Pioneer's numbers are 10X and 40X. Easy call, especially since the Toshiba is actually more expensive.
History says that the top-of-the-line Hewlett-Packard tape drive is either going to be the best-of-breed or close. The top-of-the-line HP DDS4 drive appears to be the C5685, with a capacity of 40GB and a transfer rate of 21.6GB/hour (assuming hardware compression).
For my purposes, clearly displaying a lot of text at relatively small font sizes is the most important thing I want a monitor to do. Thus I pick the only monitor PC World rates as excellent at both text and graphics, the Mitsubishi Diamond Pro 2060u. It supports 2048 × 1536 at 75Hz, a refresh speed comfortably above flicker level.
Daryll is a graphics expert and part of the team working on the Linux drivers for ATI's high-end Radeon card. He tells us that for the foreseeable future (or at least until NVIDIA gets a clue about open source) the Radeon will be the best high-end graphics card with entirely open-source drivers. So we add one ATI Radeon 64MB card.
Because this is a development box rather than a gaming machine, it's more important that a sound card be well supported with stable drivers than that it hug the bleeding edge of audio technology. The safe choice seems to be the SoundBlaster Live Platinum 5.1. ConsumerSearch's top speaker pick, rated excellent for both game play and music, is the Klipsch ProMedia 2.1.
Like many hackers of a certain age, I imprinted on the IBM Model M keyboard about 20 years ago. They have a relatively stiff travel with a sharp break and a positive keyclick that can only be described as crunchy. They inspire cult-like devotion. It's still possible to buy the real Model M, armor-plated case and all. They're not being manufactured anymore, but old stocks are still being sold. You want these IBM model numbers: 42H1292 (IBM 101-key, buckling-spring keyboard) and 1393278 (IBM SpaceSaver compact, heavy-duty 84 keyboard). They're both available from Unicomp. The dream system will get one of the 101-key PC-2 versions.
For my own use, I'll keep my original three-button Logitech TrackMan Marble. Sadly, Logitech doesn't make the original Marble any more; the replacement has a rather obtrusive wheel replacing the middle button.
There is only one possible modem for the dream system: the US Robotics V.Everything, external version. This featureful, rock-solid, reliable modem is the first choice of discriminating hackers everywhere. Rick has written an entertaining rant on the likely consequences of choosing lesser external modems, or any internal modem at all.
The floppy drive is a relic of the age before bootable CD-ROMs. Occasionally you'll want one for booting up diagnostic software. A plain old TEAC 1.44 3.5" drive will do.
Oh, yes, the software. I realize that the topic of favorite Linux distribution is a religious war, but I can't resist putting in a plug for my own favorite: KRUD Linux from Kevin Fenzi and the good folks at tummy.com. Subscribing to KRUD gives you a Red Hat base plus a monthly update, including all security fixes and a tasty selection of additional programs and tools.
We have two SCSI controllers. That's good, because we also have both LVD and single-ended SCSI devices in our parts list. Daryll observes:
LVD drives can drive the bus at 40MHz and 80MHz, whereas single-ended cannot. If you mix single-ended and LVD, the bus degrades to single-ended. So a bus with a single-ended device tops out at 20MHz Wide SCSI or 40MB/s, whereas LVD gets you up to 160MB/s.
Thus, we want to assemble the dream machine with two SCSI chains: a high-speed wide/LVD chain for the hard drives and tape, and a low-speed narrow/single-end chain for the CD-RW and DVD-ROM. We used an SM-20 from The Mate Company to convert the second motherboard channel to 50-pin narrow SCSI.
Because the hard drives are likely to be significant heat generators, we mount them with the spare internal bay between them, rather than stacking them in adjacent drive bays, to get better airflow.
The Antec case makes it possible to mount the intake fan directly in front of the hard disks. Normally, with drives in this class, the drives and the bay enclosure become uncomfortably hot to the touch; with this setup, the warmth is barely noticeable. This is a good thing because it probably extends the expected lifetime of the drives significantly. Another fan near the power supply at the rear helps pull air out of the machine. We ended up mounting a third fan because we noticed the memory chips seemed to be running hot.
We'll have two expansion cards in the machine, the SoundBlaster Live! and the Radeon. The Radeon will probably tend to run hot, the SoundBlaster not. Happily, the Radeon lives in the AGP slot at the upper end of the slot row, where the air it heats will be sucked into the two rear fans.
How does our noise budget look? IBM says our UltraStars emit 48dBA each, PCP & C says the power supply emits 44dBA and the fans 20dBA each, and Tom's Hardware rated the Silverado at 37dBA (but there are two). Applying the logarithmic-sum formula gives us 52dBA as the level of interior noise. Assuming the case blocks 8dB, that will leave us with an exterior noise level of 44dBA adjacent to the case. We can trim another 5dB or so by putting the machine desk-side.
Recalculating with four or five case fans barely nudges the second decimal place in the total. This means that in case our initial burn-in reveals a heat problem; we've got room to cool things down without making the machine substantially noisier.
Gary Sandine and John Pearson at Los Alamos Computers undertook to assemble my Ultimate Linux Box; in fact, they assembled two, one for me and one for Linus Torvalds. They solicited the vendors on our list for donations of parts, and their courage was rewarded when IBM generously volunteered $15,000 for the project budget.
I find it impressive that, after having specified it on a cost-is-no-object basis, the total system cost is so low. I tried to gold-plate as much of the system as possible and load on all the extras and accessories I could, and I was still unable to raise the total parts bill over $7,000 US.
If we discarded the most extravagant peripherals—the Klipsch speakers, the Radeon, and the DVD and DDS drives—the cost would drop to a quite reasonable $4,200 US or so. As Rick pungently observes, “People pay more than that for crap computers every day.” This design will be available for purchase from Los Alamos Computers as the ULB-200108.
And how fast does it build kernels? After make clean, the Ultimate Linux Box builds the ULB's 2.4.8 Linux kernel from a cold standing start (make -j3 'MAKE=make -j3' dep; make -j3 MAKE=make -j3' bzImage) in 2 minutes and 21 seconds flat. Sweeeet.
Eric S. Raymond is a wandering anthropologist and troublemaking philosopher who happened to be in the right place at the right time and has been wondering whether he should regret it ever since.