Follow the conventional IT media and you'll miss the new level of self-reliance and participation in Linux at companies large and small. Executives from Ernie Ball, Morgan Stanley and Ticketmaster explain the shift to “do-it-yourself”.
Without a doubt, Linux and open source are changing IT (Information Technology) at companies of every size. But how? When you read the IT magazines, go to the IT conferences and listen to the IT analysts, you get the same message you got ten or twenty years ago: vendors are in charge.
Of the 62 stories in Computerworld.com's current Week In Review (mid-November 2003), 51 stories, or 82%, are either about vendors—“Red Hat goes Live with Fedora”—or refer to vendors in their headlines—“IBM's Palmisano says US must innovate to keep jobs.” In reality, though, vendors' and customers' IT worlds are steeped in a variety of development communities. Both vendors and customers develop goods for themselves, as well as for sale and for use by the rest of the world.
Over the past year, I've been on assignment by Linux Journal to study what's really happening in the IT marketplace and the deeper roles played by Linux and open-source development in that marketplace. My first report was “How Linux Makes Companies Smarter” /article/6585, in the July 2003 Linux Journal. This second report focuses on changes in IT itself. What I've found is an increasing reliance on personal and development community initiative and the freedom and trust making that possible. In sum, what I'm seeing is a do-it-yourself movement in companies everywhere, a growth in self-reliance I'm calling Do-It-Yourself IT (DIY-IT).
Phil Moore, the Executive Director of Enterprise Application Infrastructure for Morgan Stanley and Company, explains:
Open source has lowered the threshold at which do-it-yourself is possible. You can't do everything with building blocks from vendors. They pretend they're selling you a prefab building and they're not. They're selling you pipes and fittings and stuff to put it together.
In reality, to build an enterprise, you have to have a set of experts in your IT shop who can put it all together. Certainly, historically you need a lot of expertise to get anything done, because this stuff really isn't easy to put together. But if you've been led by the vendors to believe that everything dovetails together nicely, like you see in the .Net ads, or in any major marketing campaign that promises nirvana, you've got a problem.
You always need a certain amount of do-it-yourselfness. Consultants don't walk in, deliver an enterprise and walk out, saying “call me in six months for an upgrade.” It's organic. An enterprise is changing constantly. Even the walls in your house right now are on their way to needing another paint job.
Although DIY-IT involves a reduction in dependency on vendors, it doesn't mean vendors are bad or that they don't play extremely important roles in the marketplace. It does mean that the marketplace no longer belongs to them. It means a new balance of power exists between supply and demand as does a new division of responsibilities between vendor, customer and development communities. It seems the software business is growing up.
In this report, we look at several key factors involved in the DIY-IT movement: what leadership really means, the role of the Net, the rewards of courage, the cost-savings imperative, valuing talent, where we stand, untold stories and perspective.
Looking to leaders for leadership is natural. But what about the leadership of developments that have no direct leader—developments where the leadership comes largely from within, from shared conviction and the practices that express it?
That's what we have with Linux, with free software and with open source. Linux is a development project, not a company. It is not contained by a corporate structure. Like a tectonic plate, it is held together by cohesion more than by organizational forces. Free software and open source are value systems and development methodologies. To treat them strictly as populations or as classes of goods is to miss the nature and scope of what they're about.
The Net not only supports much of what we now take for granted in the technical world, it puts everybody and everything in a position to get more connected, more informed, more intelligent. Public bits outsmart secret ones, even if secret ones still have economic and other forms of value.
Craig McLane, VP of Technology at Ticketmaster, puts it this way:
The best thing about the Internet, to me, is it mitigates tremendously the friction imposed by time and distance. You can take people who have individual passions and great talents, unite them and eliminate many of the obstacles to communications.
The Open Source community truly is global, and that matters an awful lot to us. We're an international company and we like access to very smart people worldwide as we grow—people who still are accessible and relatively close-knit.
The notion that you can take part in this community, do great things, be supported and foster this whole breeding ground of innovation is absolutely incredible.
Trying to imagine a civilized corporate or governmental organization today that is not sustained by the Net, and therefore also by open-source software, is almost impossible. Organizations everywhere also are coming to recognize that they depend on open-source values as well as open-source talent and code. In just the last year it became clear to governments around the world that their computing infrastructure needed to be built on stuff that is open, that has no secrets. They want to be able to inspect it to make sure it's sound, reliable and open to improvement.
Take Sterling Ball, the leader and namesake of Ernie Ball, the guitar string maker. You can't press SCAN on the car radio without hearing an Ernie Ball string. What makes Sterling Ball an open-source revolutionary isn't his technical chops, it's his independence—his guts. This became evident after Ernie Ball was raided in 2000 by federal marshals as part of an unannounced software audit by the Business Software Alliance (BSA), which found unauthorized “pirated” software on some of Ernie Ball's computers. The BSA still brags about the raid on its Web site. Here's how Sterling Ball described it when I interviewed him at LinuxWorld in August 2003:
A disgruntled ex-employee saw a nail-your-butt opportunity, so he called the BSA. I was sued under federal seal. There was no warning. We were raided at 10 am on a Friday. We were shut down and ordered not to touch our computers. There were armed marshals. Our employees were sitting there going “What's the matter? Is our company criminal? Are we crooks?” Then they sent out press releases....It's coincidental that they always send these out after a business is closed.
We're the number one employer in terms of manufacturing in San Luis Obispo. We're a big fish in a little pond. The headline reads, “Ernie Ball Raided for Piracy”, and the story says, “Company officials unavailable for comment”. Well, no [surprise]. I was at home. And I never say “No comment”. So, when it came time to tell my story, I said, “They came for bear and got squirrel.”
Ernie Ball went to court and paid a fine, but that didn't end the matter:
The worst thing was when Microsoft printed a four-color reproduction of that newspaper article on an executive's desk, sent it to every registered Microsoft user and said “Don't get caught like Ernie Ball—a fine company that found out just how hard it is to stay compliant. Call us. We'll give you a free audit and sell you software at 20% off.” Keep in mind that we already had downloaded the BSA self-auditing software and it didn't work. This was fear-based marketing, with government help.
Sterling Ball didn't get mad or get even—he got out:
Everybody thought I was crazy. The IT people thought they were going to get fired. I said “no”, because I've never seen any greater programming in the world than “You can't do business unless you've got an office suite on your desk.” Hey, I'm talking here at LinuxWorld because I changed my word processor. The solution everybody [at our company] uses is a cocktail of open-source stuff. Nobody showed us how to do it. We had to figure it out ourselves.
Today Ernie Ball's servers run Red Hat Linux. Its desktops are GNOME on thin Sun clients, with applications that run off a Linux server. The company time clock and security software run on Linux. The company e-mail is Ximian's Evolution, and their office suite is OpenOffice.org.
The lesson here isn't about technology. The lesson is about independence, integrity and the courage to break free of mental programming. It's a lesson about the souls of individuals, of organizations and of a marketplace that still thinks vendors are in charge, even though the success of the Net and the Open Source movement prove they are not.
Breaking free is akin to awakening, and it doesn't happen only for companies the size of Ernie Ball. Take Ticketmaster, for example. Here's Craig McLane again, speaking at LinuxWorld:
We support 8,000 clients in ten countries. In 2002 we sold 95 million tickets through channels that represented over four billion US dollars. This puts us in the top 25 of all retail Web properties—actually number two, between Dell and Amazon. So we're doing a lot of business, but on behalf of other people who have entrusted us with their business.
We have 3,500 outlets, 19 call centers and the Ticketmaster.com Web site, which does about 50% of our business. We also provide box-office solutions. If you've ever purchased a ticket at a box office, that's also a Ticketmaster system, with the same inventory bucket.
Our product and technology organization is the cornerstone of Ticketmaster as a company. We've got 250 people devoted to product and technology in an organization that has about 2,000 full-time employees.
We provide solutions and systems, but we also support those 8,000 clients. In many cases, because of the nature of the business—highly customized, highly variable traffic and all kinds of strange configurations that actually are much different from any other retail businesses—there are no commercial solutions available for what we need to do.
In fact, we are one of the first application service providers: extending the service to thousands of clients, actually writing the code, hosting the system, providing the customer service and charging a fee per unit sold. We have to build high-volume systems for very specific and peculiar businesses. Open source allows us to do [this] as well or better—at least in our experience—at half the price of commercial solutions.
That's why Ticketmaster converted the Ticketmaster.com site primarily to open-source technologies over the course of the last 18 months.
McLane showed a small spreadsheet at this point in the talk (Table 1) that outlined the cost of the company's computing needs in terms of open-source and proprietary options. He continues:
Table 1. Ticketmaster Computing Options
| Ticketmaster.com | Open Source | Proprietary |
|---|---|---|
| 400 PC-based systems | $1,000,000 | $1,000,000 |
| Operating system | $0 | $600,000 |
| Web server software | $0 | $120,000 |
| Database software | $0 | $240,000 |
| Total | $1,000,000 | $1,960,000 |
It's really all about licensing costs. We buy the the same class of machine, same configuration, from the same vendor. But we're using all open-source technologies in these areas, and we pay nothing for licensing. So you can see that we save 50 cents on every dollar we invest and get the same or better performance. And we see better support from the community than we typically get from commercial vendors.
The Web site costs, in hardware and capital development, a couple million bucks. So it's a small fractional part. It doesn't materially change the business from Wall Street's perspective or from the CFO's perspective. But it matters to us because we can use the money that's made available to employ more smart people. And that's really the key.
To Craig McLane, open-source human resources are collective as well as individual. To explain, he quotes T. S. Elliot: “No poet, no artist of any art, has his complete meaning alone”. He adds:
By providing really robust tools to our people and by pulling in more people who are interested in solving problems, who can work autonomously, but who like to have the benefit of a larger community while they're working....
[People like] Stas Beckman are doing core work for us...mod_perl 2.0. [He's] doing a lot of the work that will benefit the community, which benefits us. He's also working on database connection pooling....So we get to take the lead on doing some things that have broad general benefits for everybody. Geoffrey Young is going to start working with us. He's going to be working on things that can be done autonomously, that benefit the community but also provide immediate benefit for our business when deployed in very specific ways.
Our teams have ownership over their tools. They also don't have an excuse. They can't say that a vendor doesn't have an answer or isn't getting back on the phone. Because everybody knows that there's a community out there and you have access to the source. Everything is in front of you. So there's an accountability that's reinforced when you have source code and a community that knows so much and is so willing to respond.
You're also more motivated. When people pick their tools, the work invariably makes sense to them. They're also working side by side with the people creating the tools that we're using day by day. You can't get that anywhere else.
Once again, a company gets smart and saves money by aligning itself with its own smart employees and the development communities to which they belong.
What's different between now and 10 or 20 years ago? McLane says it's “the amplifying effects of the Internet on the power of the individual”, adding, “only a fool would ignore that”.
We read about IT brass going gaga for Linux almost every day. Ken Harris, CIO of Gap Corp., recently said he's in favor of “Anything touching Linux”. Emea Harris of Lehman Brothers said, “We're very aggressive around migrating to Linux.”
When I spoke about DIY-IT at the O'Reilly Open Source Convention in July 2003, the majority of those attending the talk (about 100 in the room) was rank-and-file IT guys, mostly from large companies. Phil Moore was one of them. One audience member said some companies feel that their own open-source developments give them a competitive edge, and they don't want to talk about it for that reason. “They don't want their competitors to know how they do it faster and cheaper”, she said.
What other kinds of stories are we not hearing, then? Here are a few, in no particular order:
Debian: “I'm seeing far more Debian than any report gives it credit for”, says one technologist working for a large vendor that has partnerships with Red Hat and SuSE. “Red Hat and SuSE may sell more, so they show up on surveys that follow sales. But in terms of actual implementation, Debian is pretty big.”
Education: at different periods during the past 30 years, companies like Digital, Apple, IBM and Microsoft have had successful programs for getting students hooked on their goods. Now those students are swimming in a sea of free software and old or cheap PCs on which to run it. Web services consultant and author Doug Kaye says “High schools and colleges are now all about open source....It's LAMP everywhere.”
The power of gravy: a number of IT people have told me that vendor relationships are valued highly, period, and always will be. “You get freebies. Tickets to games. Free dinners. Trips to conferences. A lot of people love that gravy train.”
Small consultant opportunities: one IT guy at a large company told me:
The do-it-yourself movement inside IT is lowering the barriers to entry for small contractors too. Thirty years ago, big companies went to big vendors for big solutions. That's not the case anymore. You've got small vendors and consultants with 15, 20 or 40 people going in and delivering hugely successful solutions to Fortune 50 companies.
Phil Moore adds:
When you contract for an open-source solution, all the middle layers of the contracting nightmare you get when you go through big vendors are gone. If you contract for a simple change from a big vendor, you can be in for spending a lot of money. The cheapest vendor contract I've ever been involved with is a half-million dollars. At the very least you've got two gigantic legal departments involved to begin with, just to negotiate the contract. Yet I've gone and got similar order-of-magnitude technology changes on open-source products for below five figures. This is a gold mine. This frees up a huge part of my budget. And time-wise the process is highly streamlined.
Without vendors, we wouldn't have magazines or tradeshows, to name two of my favorite things. It's important to the market's ecology for vendors to push their goods and tell their stories. The problem we've had—and still have—is a long lag between what's happening in the marketplace and how we cover the subject. And I believe that lag derives from the young ages of the industries involved. The computer industry is about 50 years old. The software industry is half that age. The Internet—which changed everything—began supporting business only about nine years ago. What we need are more stories from the demand side of the marketplace and more courage by those in positions to tell them. We also need publications that welcome those stories, with authors and editors and analysts to help tell them.
My friend Christopher Lydon (blogs.law.harvard.edu/lydon), a former reporter for the New York Times and host of NPR's “Connections”, believes what's happening in our industry—this DIY-IT movement—is profoundly Emersonian. He points to this encouraging prose from the author's seminal essay, “Self-Reliance” (www.emersoncentral.com/selfreliance.htm):
To believe your own thought, to believe that what is true for you in your private heart is true for all men, that is genius. Speak your latent conviction, and it shall be the universal sense; for the inmost in due time becomes the outmost....There is a time in every man's education when he arrives at the conviction that envy is ignorance; that imitation is suicide; that he must take himself for better, for worse, as his portion; that though the wide universe is full of good, no kernel of nourishing corn can come to him but through his toil bestowed on that plot of ground which is given to him to till. The power which resides in him is new in nature, and none but he knows what that is which he can do, nor does he know until he has tried.
We'll be doing our part here at Linux Journal. And, as always, we rely on your help as well.