Linux in the Real World

Ever since the opening of the Internet to commercial development a few years ago, many an individual has made a living for himself by leasing a connection to the global network, establishing a local network, and tacking up his siihingle as an Internet Service Provider (ISP). ISPs, from the mighty MCI down to the most humble Mom and Pop basement organizations, offer exactly the same thing—a direct connection to the Internet. Smaller businesses do this mostly via PPP connections over modems, whereas larger ISPs tend to stick to leased-line dedicated connections.

Although this might sound daunting at first, the actual process is not impossible, even for a single individual of limited resources. The most pressing obstacle in the past has been that the resources needed to offer ISP-like services—not merely PPP connections but also e-mail, Usenet, and FTP/WWW hosting—have only been available in the form of high-end (and high-cost) Unix workstations. This translated into considerable cost for the budding entrepreneur.

The other traditional problem has been that the system- and network-administration skills needed to offer such services could only be gained on one of these high-priced workstations, creating a chicken-or-the-egg dilemma, whereby if you did not already have access to these machines through work or school, your main capital investment would have to sit idle for six months to a year while you tried to learn how to run it.

The ubiquitous nature of Intel-x86-based personal computers, the strength of their networking capabilities and, most of all, free operating systems such as Linux brought this sad state of affairs to an end. With its traditions of laissez-faire development and absolutely free access, Linux offers two qualities essential to the upstart ISP. The first of these is minimal cost. The second, and the more important, is the ability to learn all the essentials of network administration on an operating system which refuses to withhold any secrets from you.

It is hard to reinforce this latter aspect of Linux enough. As an independent Internet Service Provider, your job is network management. If your machines are not routing traffic properly, you must understand why and be able to fix it. If your Usenet feed is clogging your system, you must be able to diagnose the problem and fix it. If your DNS is not spitting out IP numbers properly, it is your problem, and yours alone. Microsoft technical support will not come down and help you figure out why. Sprint or MCI will not drop what they are doing to help you out, even if you are their customer. Good consultants (other than the author) are notoriously hard to find and (including the author) are horribly expensive.

Linux is the operating system of choice for ISPs, because it fulfills the number one requirement for an ISP's OS. You must be able to diagnose problems, and you must know enough about your system's operation to be able to fix the problems which will plague you, no matter what OS you choose. This, combined with the huge range of software available for Linux, make it the hands-down winner.

This is not to say that Linux is perfect, and indeed, there are several problems with Linux in an ISP environment. This article should help by offering a roadmap, showing both obstacles and bypasses to help you in your journey. It does not offer advice on how to connect to the Internet, but rather concentrates on using Linux to offer ISP services.

Your first step is deciding on a hardware base. What kind of machines will you run? How many of them will you have? What peripheral equipment will you need? You should have good answers to all of these questions before you buy any hardware.

In the past, the platform decision was easy. If you ran Linux, you used an Intel-based or Intel-clone 386 or greater machine. With the recent enhancements to Linux for the Sparc, MIPS, and DEC Alpha, this choice is a little more clouded. Red Hat and others are coming out with full-fledged CD-ROM distributions for these higher-powered machines. While these are in the early stages of development and I do not recommend them (yet), the price-to-performance edge of these machines, especially the DEC Alpha, means you should keep them in mind as a real option, especially for upgrading your system down the road. For startups, though, x86 machines are usually the way to go.

If you are going to concentrate on dial-up business, you will need modems. Modems can be one of your worst nightmares, and you should shop carefully. However, the modem issue is not Linux-specific. With V.34 finally having settled down to a well-established standard, most modems will do the job, and many people suggest shopping for price. Nonetheless, doing your homework by reading reviews and soliciting feedback on Usenet can pay off many times over. Just think about dedicating hundreds of man-hours and thousands of dollars to modems that refuse to work, and ask yourself if the extra initial effort might be worth it.

Of course, standard PCs can handle a pair of modems at best. How do you connect all of these modems to your PC? The answer is through one of the many port-servers (also known as terminal servers) available on the open market. Port servers connect to your modems with built-in serial ports and forward data back and forth between them and your computer, usually over a network. Alternatively, you can use an “intelligent serial board” which connects the modems directly to a PC. A good review of intelligent serial boards can be found in the June, 1995 issue of Linux Journal on page 46. Remember to include a port server or intelligent serial board in your plans.

Then there is your network itself. Unless you plan to fulfill all of your requirements with one machine (usually not a good idea), you will need a network to connect your machines. While Fast Ethernet, FDDI, and ATM are all options, plain vanilla 10 Mbps Ethernet is usually the way to go. I recommend 10-base-T (twisted-pair 4-conductor cable with a hub) over 10-base-2 (coaxial wire with T-connectors and terminators). 10b2 is simply not reliable enough to avoid service disruptions as you remove machines from the network, rearrange your network with expansion, and trip over the network cable. If you decide on the 10bT route, remember to equip all of your machines with Ethernet cards (anywhere from $30 to $300 each) and buy an Ethernet hub (between $400 and $1500). The NET-2-HOWTO includes an excellent discussion of the merits of the various ethernet cards; ISA bus ne2000 clones offer an easy and very inexpensive solution.

Presumably, you have arranged an Internet connection, either through one of the nationwide services such as Sprint, MCI, and ANS, or through another regional ISP. Traditionally, this connection is done via a dedicated router. Cisco Systems makes the best high-speed routers available, but they are usually overkill for connections of T1 speed or less. Livingston routers are a particular favorite among the Linux community, and other options also exist. An increasingly popular option is to make your own router out of a Linux-based PC, using a T1-interface card from a manufacturer like Enhanced Technologies. If you have an ISDN connection, then you can use your Linux box as a router too, through an ISDN terminal adapter from 3COM, Boca, Motorola, and others.

Finally, you have to decide how many Linux machines you will need. This all depends on how you have designed your network. There are two big mistakes that people make. On the one hand, some assume that every little network function needs a dedicated server. “We have to have a dedicated DNS, a dedicated web server, a dedicated FTP server, a dedicated mail server...” This can become very expensive, very quickly, and unnecessarily so. On the other hand, I have seen people make the mistake of cramming news, mail, PPP, and web service all on one machine, which is so slow that their customers leave as fast as they come in the door.

What hardware will younee and how should it be connected? How much will it cost? Through the rest of this article, we'll look at the service you can offer as an ISP and what kind of setup you'll need to do provide it.

The Linux kernel only offers the basics of an operating system. You must chose one of the many Linux distributions to use on your system. ISPs have a number of specific concerns which might not coincide with the general user's, and foremost among these are upgradeability, coherence, and network integrity.

Upgradeability is essential to the ISP because of two competing demands. First, you will need to offer the latest and greatest solutions. The Internet protocol suite of services is constantly growing, and the reason we have distributions in the first place is so that we all don't have to port, clean up, and install the various software packages on our systems by ourselves. Secondly, your service must be reliable. This means that when you do want to upgrade your system, taking it down, formatting the hard drive, and installing a new distribution is not a very good option.

By coherence, I mean how well the various components of your system fit together. For example, installing Wietse Venema's TCP wrapper (/sbin/tcpd) is not a trivial exercise, mostly because you must coordinate it with the various weird features of your network daemons. Slackware and most other Linux distributions come with tcpd built in and ready to go. A coherent Linux distribution saves you a lot of effort, as you need not fix the mistakes in the distribution.

Finally, you will live or die by the quality of your network functions. Sloppy and/or buggy compilations of network utilities, non-functioning daemons, or generally inferior network services are unacceptable in your distribution.

New distributions are usually plagued by the latter two of these problems, and this was the case with the early releases of Red Hat. However, Red Hat is very easily upgradeable and has been steadily improving the general quality of their product. Additionally, if you are brave enough to try Linux on a Sparc or Alpha, then Red Hat is probably the way to go.

Coherence and strength of network services have always made Slackware a particular favorite of mine, and I have gotten a lot of mileage out of it. It saddens me, therefore, that the persistence of bugs in the distribution and, most of all, the impossibility of upgrading a Slackware system are causing it to drop out of favor.

My own bets are laid on Debian, a package maintained much as the kernel is developed—by a team of people over the Internet. Debian is still gearing up, and there have been problems with one CD-ROM version of it, but I think that it holds the most promise.

Speaking of CD-ROM problems, get ready for them. Many times I have run into people who are having networking problems with a CD-ROM version of Slackware, to whom I recommend reinstalling the “N” section (networking) from an Internet source. More than 90% of the time, this solves the problem. We have a T1 connection to the Internet, and although we keep a CD-ROM version of Linux lying around for safety reasons, I prefer installing directly from the network, because if one vendor's CD-ROM version of Slackware is broken, only their customers will know it, but if SunSITE's copy of Slackware is broken, the whole world starts yelling.

Really, choosing a distribution depends more on what you're familiar with than anything else. If you have experience with Red Hat, for example, you will want to think seriously before selecting a different distribution, as the pain of learning a new distribution might be greater than the advantage to be gained.

Your major task after installing your new distribution is to set up your network and get packets flowing. This is not the topic of another article—it is the topic of a full-length book! Fortunately, the Linux community is very responsive to the needs of its customers (i.e., itself), and one of the main reasons to use Linux is because of the documentation. You should read the Net-2-HOWTO as an absolute minimum, and a copy of Olaf Kirch's Linux Network Administrator's Guide (the “NAG”) should sit on your desk within easy reach.

Of course, getting packets flowing is just your first concern in establishing your network. There are 6 main topics which you should consider: connecting customers, DNS, mail, news, Web/FTP, and network security. We will examine each of these in turn.

The Serial-Line Internet Protocol (SLIP) was a ground-breaking development in computing. Then again, so was the Apple II. While still in favor in certain circles, SLIP is dead and should be recognized as such. Its successor, the Point-to-Point Protocol, (PPP) is the wave of the future. (Okay, it was the wave of a year ago, but you get the point.)

SLIP is firmly entrenched in second place to PPP for several reasons. First of all, SLIP is only capable of serving IP traffic, whereas you can run virtually anything over a PPP link, including IP, AppleTalk, IPX, and others. Second, PPP's Link Quality Management (LQM) functions give you a solid connection by running it at the proper speed, regardless of electrical interference or other line noise. Finally, the newer versions of PPP will have world-class authentication, enhanced LQM, and other features, such that PPP will continue to pull away from SLIP in terms of quality.

Linux comes with both SLIP and PPP built in. Your kernel needs to be compiled with both SLIP and PPP support. Remember this when you install your system, or you will rack your brains trying to figure out why one or the other is not working when you try to use the missing one.

You will want to read the PPP-HOWTO, as it is invaluable in understanding how to make PPP work. You will need to decide a number of things, such as whether to give customers shells from which to invoke PPP or make PPP their default login program.

Furthermore, how will you do the accounting for their time on-line? The answer to the previous question might affect this one. Adam McKee's BBS-Util might be helpful, as might a number of other packages. Ask around, look around, or do what we did—write your own.

Getting PPP/SLIP to work is a medium-sized job, and you will want to test your setup via modem with as many different OSs as possible. Win 3.11, Win95, OS/2 and OS/2 warp systems need to be able to dial in, and you need to test all of these systems to make sure that you can explain to your customers how to do it. Writing a HOWTO for each OS as you test it is a good idea. And don't forget your Macintosh customers! They suffer under the oppressive boot of hegemony as much as MS Windows users do, and they usually make good customers, as they are willing to pay good money for a product that works.

A good decision might be to put your PPP customers on the same machine as your mail server, making it easier to maintain only one /etc/passwd file for the entire operation. pppd will use username/password pairs from the /etc/passwd file, so throw that pap-secrets file away!

PPP is fairly time-consuming to set up, and if you decide to offer non-IP services, this, too, will take time. However, usually once it's up, it's up, and you can go on to other problems. Just be sure you have backups!

While computers understand 32-bit IP addresses, you and I don't. My e-mail is tlewis@cheney.net, not tlewis@204.214.16.150 (although that works, too). The Domain Name System is the glue that holds together domain names and IP numbers.

You do have an Internet domain name, don't you? You will need one, and once you have it, you will need a Domain Name Server (DNS) in order to use it. DNS setup is fairly straightforward, and I have written a mini-HOWTO on the subject, which will be available from the LDP archives by the time you read this. [FIXME: More information coming here]

DNS places a very light load on a machine, and your DNS server can serve in other capacities, too. Some people put it on their PPP server, some on their news server. We have two: one on our WWW server and the other on a 386-40 dedicated to DNS. To each his own.

E-mail is probably the most useful and the least appreciated aspect of the Internet. Once you have tired of the pretty pictures of the WWW and want to get some work done, e-mail is the tool of choice. Nonetheless, people neglect it too often, at their peril.

If your news server goes down, you will get some gripes, and if your web server goes down, you will get some calls, but if mail goes down, Annie bar the door! Your POP server (the program that lets customers use programs like Eudora to read their mail) usually works out of the box, but your SMTP daemon (such as sendmail) is what does the heavy lifting, and this is where your efforts will be directed.

The NAG includes a good section on mail, and the Mail-HOWTO is also a good starting point. Mail is another of those services with which, once you have it working, your problems are over, but you will spend time getting it working properly. If customers want UUCP, mailbots, or mailing-list support, this means additional work, and you should consider the amount of additional work before knee-jerking into a “Sure, we can do it!” response.

Usenet, the global electronic news system, is distributed mainly over the Internet, although certain UUCP networks and others also offer it. Simply put, any message written by anyone anywhere submitted to public Usenet groups (1000, 4000, 10000—how many are there?) will end up on every Usenet server on the planet. Last I heard, a full Usenet feed (i.e., accepting every group) runs about 400MB per day, so you probably don't want a full Usenet feed coming in over your 56kbps connection.

The News-HOWTO goes over the various options for news servers (programs which accept deliveries, organize the articles, and spit them back out to customers as they request them). Pick one, learn it, and stick with it. At 400MB per day, news is an inherently dangerous thing, kind of like having a water main feeding into your sink, and you can lose control of your news server without a whole lot of effort, filling disks and ruining your day.

News also exacts a heavy toll on the machine that serves it, consuming large amounts of RAM, CPU cycles, and disk space. At the very minimum, you will want your /var/spool/news directory on a separate physical disk. Usually, it is a good idea to have a separate news server, which perhaps also serves as a DNS. Our 486-DX66 groans slightly under a full news feed, and a Pentium might not be a bad choice. Then again, you might not want to run a full news feed, in which case a 486DX-33 might do the trick.

Slackware and most other distributions come with Washington University's wu-ftpd anonymous FTP server built-in, preconfigured, and ready to go. Go to /home/ftp/pub, start throwing stuff in there, and you're off to the races. If your customers want to be able to put up files for FTP on your server, then you might have some work to do.

The WWW, despite all the hype, is not especially difficult to implement, either. Apache, a derivative of NCSA's httpd, is a favorite Linux tool and comes with plenty of neat gizmos. Virtually all web servers are capable of serving information from a public_html directory in a user's home directory, accessible as www.foobar.com/~username/. If you decide to do this, you will want to put your web daemon on the same machine as your users' home directories, which may also be your mail machine. If this, along with PPP, is too much, break out your PPP server and leave mail and web on the one machine. Plus, you can have users access this one central machine via multiple PPP servers (even via multiple remote PPP servers across your greater metro area), instead of having each server duplicate these functions.

ISPs offer access to their networks to people they usually have never met. As such, you will need to keep a constant eye on the security aspects of your system. Start with a good introduction to network security, such as Cheswick and Bellovin's Firewalls and Internet Security, which served as my introduction to networking in general.

A good place to start is with a packet filter on the router connecting you to the Internet. A good packet filter and careful password management are two small steps that will put to rest 90% of your security concerns. The most important step you can take with Internet security is to understand it and to use the tools (like TCP wrapper and packet filters) at your disposal.

The above descriptions cover the high spots of all the issues which you will face in starting up an ISP business using Linux. It is doable; we at Cheney Communications and countless other ISPs are living proof of this.

You will have problems with your network. You should get used to that fact now. Fortunately, the good people on Usenet in comp.os.linux.networking (myself included) are always ready to help. The Linux Documentation Project is an invaluable resource when trouble arises. It is even more useful before trouble arises!

By now you should have a pretty good idea of what equipment you will need in order to start dishing out IP services. Are you going to offer limited Usenet and e-mail to a few businesses? An ISDN connection and a pair of 486DX-66s should do the trick. Are you starting up a full-service ISP for dial-up and leased-line services, with a full news feed and commercial web hosting? Three Pentiums (dedicated news server, dedicated PPP server, and a mail/WWW machine) and a router might be a good start.

You need to make sure that you have all of your ducks in a row as far as the business end of the operation. How will you keep track of billing? What will you charge your customers? For what services will you charge extra?

Is Linux the right operating system for you? If you have experience with Berkeley-style systems, maybe NetBSD would be a better choice in the short run. If you are in a corporate environment and are setting up a network for your business, maybe you can spend the extra money for the technical support of Solaris or SCO. Then again, Linux has, in my opinion, a better range of services than BSD, and technical support for Linux is available from SSC and others, including several LJ advertisers. Instead of being locked into technical support from one vendor, you have a choice. And why would anyone run SCO?

You should not be under the illusion that becoming an ISP is easy. News will go haywire on you again and again if you are not an expert (you will become one or die trying). The business can be very competitive in different regions, and your dreams of wealth and glitzy nerd-dom might die the hard death of too much work and too little money.

Before investing the time and money in starting up an ISP business, you should be sure of your ability to do it. If in reading through the documentation mentioned above, you had trouble understanding it and are not confident in your ability to pull it off, maybe jumping right in is not the best decision. Again, Linux comes to the rescue. You can get a dial-up account with another ISP and set up a trial system on your PC at home. If you cannot handle a partial news feed, your own mail server, a DNS, and a web daemon, keep hacking at them until you can, and then reconsider starting up. I can think of few deaths worse than being condemned to run a network when you don't know what you're doing, especially if your life's savings are riding on it.

Finally, to all of you startup ISPs out there, I wish you good luck. With skill and hard work, it can be a rewarding business, and you get the satisfaction of meeting interesting customers and introducing them to the Internet. With the tools which Linux provides, there is no reason why you cannot build a first-rate network (and hopefully this article will help, too). Simply be aware that you are not alone in the ISP market, and your competitors will always be breathing down your neck. We might be one of them.

Happy Linuxing!