NEC Corporation's Express5800/320La is the first commercially available general-purpose server offering hardware fault tolerance for Linux. Intended for standalone use or as an element in a high-availability cluster, this server features redundant CPUs, memory, disk, I/O and power. Hardware failover circuitry allows normal operation to continue despite loss of any single unit. Hot-swap capability extends beyond the usual power supply and disk. If a CPU, RAM or I/O card fails on this system, it is isolated and processing continues without interruption. You may replace the failed item at your convenience, without taking the entire system down. This could provide significant cost savings, for example, to a company needing servers that are always up, at far-flung locations where technical support might be hours away. Applications require no high-availability modifications to use this system as a standalone server, nor do they require failover scripts and planning.
Thousands of these servers have shipped with other operating systems, and now Linux is available on them. A stock Linux kernel provides too little error detection and recovery for this mode of operation, so NEC has added extensive hardening. SCSI, Ethernet and Fibre Channel drivers and support code in particular are modified to provide fault detection and failover. NEC's currently shipping kernel is based on version 2.4.2, with backports of some later changes. At the time of this writing, NEC was reviewing and documenting its kernel changes for a planned public release, perhaps through OSDL's Carrier Grade Linux Project. NEC is a founding member and a sponsor of OSDL.
The Express5800/320La has four Pentium III 800MHz processors arranged in pairs together with RAM and other circuitry, in two hot-swappable CPU modules. Both modules run the same instructions in lockstep, checking each other's outputs. A failed unit is isolated almost instantly, allowing processing to continue with no observable interruption. Monitoring software keeps tally of recoverable failures, such as ECC corrections to memory output, allowing diagnosis of certain incipient problems prior to larger failures. The stock filesystem on this server is ext2.
A total of three pairs of internal 18, 36 or 73GB drives may be installed and configured in RAID-1 pairs, providing up to 219GB of internal storage. An NEC S1200 RAID array may be connected through a redundant Fibre Channel, providing up to 2TB of additional fault-tolerant storage.
Two PCI modules feature dual identical sets of PCI cards. The base unit has one Ethernet card in each module. Both cards are connected to the same network; when one fails, the other takes over using the same MAC and IP addresses. All modules and power supplies plug in to a passive backplane.
Hardware watchdog timers look for system failure—for example, a system lockup due to kernel panic—and may be configured to initiate an automatic reboot either to full run mode or to diagnostic mode.
This server is large, measuring 14“ wide by 21.5” high by 27.5“ deep and weighing about 150 pounds. An 8U rackmount version also is available. A three-year warranty is included. Telephone support is provided by NEC during regular business hours.
Unpacking our review unit's well-traveled shipping crate, I observed a warning sticker on the case saying “Exercise caution when handling the system to avoid personal injuries.” NEC isn't kidding. The help of a strong coworker was needed to lift this thing gently out of its shipping crate and place it on the floor. Our demo unit had dual Seagate ST318404LC 18G SCSI drives, 1GB of RAM and two Ethernet cards.
Internal assemblies look to be well made, with no tools required for removal and replacement. Better labeling of the units would be nice, though. Fans are located in the removable units, so you don't have to take one of these servers down to replace a failing fan. Even the power cords are redundant. This allows powering the server from two independent power sources, not to mention letting the harried system administrator unplug a cord to untangle it without interrupting anything.
After pressing the power switch, located under a hinged plastic protective lid, a chorus of cooling fans kicked in with a hearty whoosh measuring 63 dBA at the front panel, 74 dBA at the back. The front panel LCD status monitor showed diagnostic messages and LEDs flashed. After about two minutes, the system completed a power-on self-test and booted up into NEC Linux, which is based on Red Hat Linux 7.1.
The popular bonnie++ disk test program was the first thing we tried on this system. Immediately upon bonnie++ startup, the fault light on one CPU module came on. The test completed, as expected, but it seemed prudent to correct the problem with the server. An NEC engineer reached over the support line had us run a few tests, and then suggested that the passive backplane had suffered mechanical damage, possibly in shipping. The backplane isn't hot-swappable. He wanted to examine it, so we arranged an exchange of servers. The new server arrived in good time, booted up and survived bonnie++ quite nicely.
To test networking recovery, I unplugged the Ethernet cables from each of the two Ethernet cards, one at a time. Ping indicated a few packets were lost, but overall communication was maintained. An rsync between the test unit and another server completed without error, despite continual unplugging of alternate cables, one at a time, with several seconds of overlap while both were plugged in.
While running bonnie++, I disconnected power to each CPU module and then reconnected it. In each case the CPU module came back up after running diagnostics for a couple of minutes. The disk benchmark results were unaffected.
The best benchmark is the load you plan to run, and a lot of benchmarks can be used. Each captures some limited view of what a system can do. As benchmarks become popular, manufacturers tweak hardware and installations to optimize benchmark results, and results thus become less applicable.
At Linux Journal we perform a rough evaluation of servers based on results of bonnie++, kernel build and the PostgreSQL regression test. These are run several times and the results averaged. We compared this server to a spare generic server having a single Athlon XP 2100+ processor using two different disk configurations, a single IDE drive and IDE hardware RAID-5. The results are shown in Tables 1–5.
The NEC unit held up well against the IDE RAID system in I/O tests, beating it handily in block output and block rewrite and narrowly in block input. In creating large numbers of zero-size files, the generic machine having the faster CPU won. The NEC server fell between the two configurations of the generic machine on elapsed time in the PostgreSQL regression test, while devoting a higher proportion of its CPU time, though at a lower load average—a measure of number of processes ready to run but not running. The kernel build test saw the NEC unit fare not nearly so well. Perhaps this is because compiling a kernel is computationally more intensive than the other two tests, but it doesn't adapt as well to multiple CPU operation, even though multiple processes are at work.
We did dry runs of these tests, and the relative loads on the two CPUs swung widely back and forth during the bonnie++ and kernel compilation tests. The PostgreSQL test showed fairly stable load allocation between the two CPUs. This is not entirely surprising, as the PostgreSQL test splits processing between client and server, offering better possibilities for distributing load between two CPUs.
Overall, the NEC machine was not blazingly fast as compared with our generic computer, a middle-of-the-road machine of recent vintage. But, it held its own nicely. The NEC machine's claim to fame, in any case, is its fault tolerance. In this, our generic machine offers not much comparison.
The NEC server comes equipped with NEC Linux, derived from Red Hat 7.1. It is text-only at the console, with no X server included. X libraries and clients are furnished for use from remote displays. Installation was reasonably complete, although I found somewhat annoying the absence of certain programs I rely on, such as procinfo.
Software re-installation is accomplished using Red Hat's kickstart method. When we tried it, re-installation from the provided CDs went without a hitch.
Some of the installed dæmons are very old releases and contain serious security holes. Examples include Sendmail 8.11.2, Apache 1.3.19, OpenSSL 0.9.6 and OpenSSH 2.5.2p2. We were unable to learn of concrete plans to ship newer versions. John Fitzsimmons of Aspire Communications, who served as our liaison with NEC during this review, indicated NEC expects its customers will customize the distribution to their own liking, clearing the final setup with NEC prior to deployment. Upgrading these things is likely to be the least part of the customization. Notwithstanding, an upgraded NEC Linux may be available later this year. It may contain an X server; we hope it contains security upgrades.
NEC supplies extensive proprietary software for configuration and monitoring of the server. This allows setup and configuration of the redundant hardware and monitoring and reporting over SNMP or other means to remote systems.
At press time, quite a bit of work remained to be done on the software load. For the price, starting at $24,000 US for the server reviewed, it would be nice if the customer did not immediately need to begin significant security-related upgrades.
We have some concern about the future of the kernel on which this server's fault tolerance depends. There have been many changes to the 2.4 kernel since 2.4.2, the version this server's kernel is based on. We were unable to obtain a clear reading of NEC's plans with respect to carrying forward its changes to later 2.4. Although there is a plan to carry forward these changes to kernel version 2.6 and beyond, and to offer Linux on other NEC fault-tolerant servers, there seems to be no announced dates associated with this plan.
NEC told us they are firmly committed to releasing all changes to the public under GPL. Until this happens, some time after this article goes to press, there is no way to evaluate how extensive NEC's changes are or what their likely fate might be, as to integration into the mainline sources for the kernel.
Our overall evaluation is favorable, but with caution. This server has more than adequate processing capacity for many applications. It is nice to be able to replace failed hardware without bringing down a system. It is nicer still to have hardware that takes itself out of service, without interrupting anything, pending a convenient time for repairs. That said, we'd advise a company considering purchase of a number of these servers to ask careful questions concerning ongoing kernel development. Also, ask about NEC's demo or “try 'n buy” programs under which you may be able to obtain one of these machines and test it hard against its intended application.