Sometimes the best lessons come from stories where almost nothing goes wrong.
“All happy families are alike; each unhappy family is unhappy in its own way. All was confusion in the Oblonskys' house. The wife had found out that the husband was having an affair with their former French governess, and had announced to the husband that she could not live in the same house with him.”—Leo Tolstoy, Anna Karenina
Stories are about problems. That's what makes them stories. They don't start with “happily ever after”. Properly equipped with interesting causes for unhappiness, they tease us toward a resolution that arrives after dozens or hundreds of pages. That's how the Oblonsky family made great literature.
The Saugus Union School District is no Oblonsky family. It's too happy. Sure, they had problems or they wouldn't have migrated to Linux. But they did it fast and with hardly a hitch. Not great material for Tolstoy, but perhaps a useful example for similar organizations planning the same move.
Being both an educational and (after the migration) an open-source institution, Saugus Union is eager to share those lessons with their communities. So, after I asked in a recent column for migration stories, the first person to respond was Jim Klein, Director of Information Services and Technology at Saugus Union. And here I am, playing Tolstoy for the School District. That's a little lesson in PR for the rest of y'all.
The Saugus Union School District is a good-sized public school system, containing a total of 15 schools and office sites serving 11,000 students in the southern California towns of Saugus, Santa Clarita, Canyon Country and Valencia. Although the district is regarded as an exemplary public school system, it's also bucking for leadership as an exemplar of resourceful and independent IT deployment and operations. That's why the top item on its Web site is “Open Source Migration”, a series of essays explaining the project and passing along wisdom for other schools.
Old-timers can guess what the district was migrating away from when Jim Klein talks about moving from one NOS—network operating system—to another. The NOS label was invented by Novell back in the 1980s. It was a positioning statement, against Microsoft's personal operating systems.
Jim writes:
When we first decided to use Novell solutions for our primary NOS, it was really a no-brainer. Microsoft's Windows NT was the only real alternative (sorry to those of you who were LANtastic fans), and it didn't scale well for our 13 (at the time) locations (I won't even go into the reliability issue, because I'm sure most of us remember the days of weekly, scheduled reboots). Over the years, we have continued to upgrade and stay current with Novell solutions, all the while giggling as we read of the pain and suffering in Redmond's world.
They kept up with what was happening in Redmond, of course, because they used Microsoft Windows on plenty of desktops, even if they kept it off the servers. Also, Jim adds, “Let's face it, Novell wasn't winning any popularity contests.” This is when they were learning about what happens when you're stuck inside a vendor's slowly depopulating silo.
Jim adds:
Then a funny thing happened—Novell acquired SUSE in January 2004 and announced shortly thereafter that it would be moving all of its services to Linux. We had taken only a casual glance at Linux up until that point and were seriously considering Apple's Mac OS X server as a possible migration option for some of our services. With Novell throwing its weight behind Linux, especially as an enterprise server platform (instead of an application-specific server, as Linux is so often relegated to in the media), we decided to take a more serious look.
Because they wanted what they were accustomed to getting from Novell—training, a choice of applications, documentation and support—they quickly narrowed their choices to SUSE and Red Hat. Jim continues:
Because of our Novell background, our first choice was to look at SUSE. Novell was more than happy to provide us with CDs, and although we knew little of SUSE short of vague references, we went forward with our evaluation. After running the installer several times (before we got it to take), we looked at the basic functionality of the system. We really didn't like the “jello-like” interface very much and had many issues getting some of the most basic functions to work. So it was off to the bookstore.
We knew from our research that SUSE was the number-two Linux distribution on the market, so we were quite surprised to find zero, that's right, zero books on SUSE Linux. The best we could find were vague references in more generalized Linux documentation. Red Hat documentation, on the other hand, was in abundance and on a variety of topics of interest. So we bought a Red Hat book, which had a free Fedora DVD in it—Red Hat: 1, SUSE: 0. Fedora installed on the first try, and with the help of some good documentation, we were able to get basic services working—Red Hat: 2, SUSE: 0. We explored more advanced functionality, both desktop and server-oriented, and found that most Web resources were, once again, Red Hat-oriented. We were able to get Fedora to do just about anything we wanted—Red Hat: 3, SUSE: 0.
But we hadn't given up on SUSE yet. Armed with a laptop, loaded with both SUSE and Fedora, we headed off to Novell's Brainshare 2004 conference in early April. Here we talked to everyone about every topic of concern. We gleaned all we could about Linux in the enterprise, spoke to techs about our concerns, looked at Novell's solutions and so on. We spoke to HP about our servers, explaining our concern over Linux compatibility with our older machines. They recommended Red Hat. We looked at Novell Nterprise Linux Services and discovered nothing unique about the implementations, other than that they were standard open-source apps installed in strange locations. We heard promises of real training programs somewhere down the road and that documentation would be coming soon. By the end of the conference, Novell had convinced us of two things: 1) Linux is, in fact, ready for the enterprise, and 2) that we didn't need them any more. (Okay, that's a little harsh—we are still using Novell GroupWise—on our Red Hat servers.)
The next step was what Jim calls “trial by fire”: installing Linux on all the staff laptops and running “solutions for everything we do on a day-to-day basis”. After a month of “self-induced pain and frustration”, they were well-conditioned for off-site RHCE (Red Hat Certified Engineer) “boot camp” training. They also accumulated piles of books and other documentation and set to work evaluating open-source replacements for the applications they had been running on NetWare. Jim adds, “Our goals rapidly evolved from potentially using Linux for some services to definitely using it for several services to 'can we use it for everything?' to 'wow, I think we can use it for everything we do.'”
Jim's advice: “...it is important to establish, well in advance, which services you need to provide, and what solution will provide said services. In some cases, options may be a little sparse, while in others, myriad. In either case, good documentation and research are critical to any implementation.”
Jim's use of the term services may seem innocuous, but it originates in Novell's intentional shift of the network paradigm in the 1980s. Before that shift, every network was a silo of proprietary offerings standing on a platform of “pipes and protocols” with names like DECnet, WangNet, OmniNet, Sytek, 3Com, Ungermann-Bass, Corvus and IBM's Token Ring. With NetWare, Novell provided the first network operating system that would run on anybody's pipes and protocols and also on anybody's hardware. As a platform, NetWare hosted a variety of network services, starting with file and print. Craig Burton, who led Novell's NOS strategy, called the new paradigm the “network services model”. Services included file, print, management, messaging and directory, among others, eventually including Web. This is the conceptual model by which we still understand networks today. It's also one in which Linux makes a great deal of sense—and why NetWare isn't too hard to replace.
The main services Jim and his crew wanted to support—directory, file, print, Web, messaging (e-mail), DNS/DHCP and backup—had Novell offerings that easily were replaced by OpenLDAP, Samba, Netatalk, Apache, BIND 9, dhcpd, Squid and Bacula (“dumb name, great solution”, Jim writes). The only remaining exception was Novell GroupWise 6.5, which lives on as a proprietary application running on Linux.
They deployed gradually, starting with nonessential edge servers and working their way to core servers and services:
We updated a Web server at the district office first and gradually added services to it for testing purposes. Then, we updated the Web, proxy and DHCP servers at two school sites. We added Samba to the servers so that Webmasters could update their sites. Then we convinced an administrator to let us install Linux on 30 laptops in a wireless cart. We learned a great deal by starting small and building up to more and more services, and the laptops taught us how to “script” the installation and rapidly deploy through the use of Red Hat's Kickstart utility. Finally, it was summer, and it was time for the bold step—full migration of 14 sites totaling 42 servers in six weeks.
They deployed everything at the server end, including automated backups for multiple PC platforms, in four weeks. Then they went out to the mass of clients throughout the school district:
When the office staff returned and were given their passwords (we had to change them as we are now on a completely different authentication system), they went right to work. We proceeded busily to remove the Novell software (except GroupWise) and join the new Windows domains (on the Samba servers) on our 3,000 or so Windows machines in our school classrooms and to update aliases and so forth on about 1,000 Macs....
When all that was said and done, we were pleasantly surprised by how smoothly the transition went. While our 800 or so users (and 11,000 students) may know that we are running Linux, it is relatively transparent to them. The Linux servers offer no indication that they are running Linux. To the Windows machines, they look like Windows servers. The Macs think they are Apple servers. Everything just works. Sure, we were in a continual state of tweaking for a while, which was understandable under the circumstances, but we did not (and have not) had a single “show-stopper” of a problem.
The dollar savings weren't small, especially for a school system. Nearly $54,000 US in licensing fees to Novell, plus $50–$200 per desktop workstation. Less measurable but even more gratifying are the ongoing time and hassle savings:
We are now able to install software, even if it has a GUI installer, remotely, which has saved us a tremendous amount of time. Software management and configuration is not only consistent, but accessible and easily modified, as opposed to being hidden away somewhere in an obscure directory object, registry entry or other mysterious location. In addition, the myriad of management and configuration tools that were required to manage the servers has been reduced, for all intents and purposes, to one. And, thanks to the Red Hat Network, we now know, in an instant, the status of all of our machines and what patches are needed and are able to schedule automated updates district-wide at the click of a mouse.
Perhaps the most interesting benefit we have enjoyed has been our newfound ability to modify solutions to meet our needs....We have, on numerous occasions, changed the way a script works or added functionality to a software package. For example, we use the idealx-smbldap Perl scripts to add, modify and delete Samba accounts from the LDAP directory. These scripts, however, did not offer the ability to add such attributes as a user's first name or title, which we needed for some of the Web applications we are using. So, with absolutely no Perl experience (although reasonable scripting/programming experience), we were able to add this functionality to the scripts and enjoy the new functionality immediately.
I was surprised that they deployed first on laptops, which are notoriously less “white-box-like” than desktops. Sleep, for example, has always been an issue.
Jim said:
We used HP NX5000s mostly, quite a long time before they started shipping SUSE on them, however. We also used NC4000s and NC6000s. We put Fedora Core on all of them, and do our installs via Kickstart. The big benefit of Fedora is that we can host a local yum repository and mirror Fedora updates (as well as other sites), which makes it easy (and fast) to distribute software and updates, through Red Hat's up2date. We don't like SUSE very much, because of the way it litters all the files all over the filesystem. It adds an extra step when you are trying to find help, as you first have to figure out what SUSE did with all of the pieces.
Sleep still doesn't work right. There are some nice kernel patches to make them hibernate, but they are a bit of work to install. We couldn't get built-in 2.6 hibernate functions to work either. This is, by far, our biggest headache with laptops. We have two batteries in all of ours, though, so we can keep them running for the day with relative ease.
On the other hand, the laptops running Linux are working great. And we've had no problems getting users to adjust. In fact, the only instruction we've offered is, “The little red hat in the start bar is the same as the Start button on Windows”, and “Firefox is your Internet browser.” They've been fine with all the rest. In fact, even trainers we've brought in from outside have had no problem adjusting to the machines and completing their tasks.
Craig Burton says “There are always two kinds of problems, technical and political. And the technical problems are usually easiest to solve.” Jim told me, “The biggest help we got from Novell was political, as they added credibility to open source through their name and industry recognition.” But, he added, “We encountered no political problems (and) almost no resistance because we came in fully informed, with all the right answers.”
I asked where he went for help during the migration. Jim replied, “Actually, Red Hat and the Web were our sources. RHCE boot camp got me up on the enterprise side of things and the Web worked for everything else. I was surprised at how much help I got from SourceForge forums and the like—even from the programmers themselves. I put my techs through Linux Professional Institute boot camp. One will attend RHCE in the Spring.”
I told Jim I often hear that, at large companies, migration is a trade of licensing costs for personnel time. Was this also the case here? “I suppose first year, you could say that”, he said. “If I consider cost in terms of our salaries and the amount of time we put into learning and doing, training fees and support fees, you could say we broke even. But then, we consider learning and research part of our job description. Outside of salaries and time, actual cash outlays were only $6,700, and savings are $50K+ per year, so I'd say we came out ahead.” Today, the district is running Red Hat Enterprise Linux 3 AS on 31 servers, and Fedora Core 1 on 11 older servers that don't meet the minimum hardware requirements for the Enterprise product.
What were the licensing fees for exactly, I asked. Jim replied, “We were a Novell shop, so it's almost all Novell fees. Generally, it's $3 a kid for Novell ed licenses—we have 11,000 students. The rest would be Veritas Backup Exec maintenance, Surf Control and so on.”
When I asked about remaining problem areas, for higher-level application migration, he said:
The problem with that move is compatibility with some of the multiuser educational software we use. Quarter Mile Math, Follett Library Automation, Renaissance's Accelerated Reader, Scholastic's Reading Counts and Orchard Software don't have Linux clients. We have pretty healthy investments there. We have experimented with Follett under Wine, and found that we can make the classroom portion work, but have not, as yet, looked at the others.
I asked about adoption prospects at the desktop level. “Several site administrators have expressed an interest in Linux desktops as an avenue for acquiring more machines for the same money, that is, to pay less Microsoft tax”, Jim said. “Most of the immediate impact has been an increased awareness of what's out there in open source. They use the Linux laptops for training and learn that they can use the same applications on their existing machines for free as well. Right now we have multiple sites experimenting with open source on Windows and Mac OS X, via OpenOffice.org, The Gimp and so on.”
As for commercial educational software vendors, Jim adds:
We've seen a fair amount of interest. For example, Follett server already runs on Linux, and we helped Quarter Mile get its Java-based server to run on Linux as well. I believe Scholastic is using a Java-based client now, which would require minimal tweaking. Better support will probably require pressure from a few good-sized districts. As we see upgrades coming, we try to force the issue a bit.
Finally, I asked him if his experience offered lessons for business enterprises. He replied:
I think the biggest thing is that Linux can be done successfully, on a multisite enterprise scale, and that Linux truly is enterprise-ready. Most of what they hear from the Microsoft camp is simply inaccurate or incomplete analysis. We've already recouped our costs, and more, and are thrilled with performance, reliability and security. Add the fact that “patch management” doesn't have to take up an entire salary, and you'll find that there's more time for innovating and less required for maintaining. I've rebooted my servers once since last September, and it was because I wanted them to reboot, not because they needed to or did it spontaneously on their own.
If you want to know more, I'm sure Jim will keep reports current at the Saugus Union School District Web site (www.saugus.k12.ca.us). The story might not be worthy of Tolstoy, but it might be worth a lot for the thousands of other school systems and mid-sized enterprises planning similar moves.