freeVSD Enables Safe Experimentation

I work in an environment where software developers and system administrators with varying capabilities aggressively and routinely use Linux. We frequently explore new applications but sometimes hesitate to actually initiate an installation because we lack confidence in the software. Our concern is that rogue software might disrupt essential services on production servers. In extreme cases, it is even possible that a poorly written installer might corrupt a workstation's operating system installation.

Typically, implementing even a slightly complicated application that interacts with the web server requires installing the application, adding a new user (for suid operation), adding lines to httpd.conf, restarting the web server and creating and manipulating files in root-owned places like /etc or /usr/local/. All of this has to be undone if we later decide not to put the system into production use. While uninstall scripts can assist in this, these scripts could fail, leaving the system in an indeterminate state.

freeVSD is a GPL product initially conceived to enable an ISP to provide virtual server hosting. It can also transform a stock Red Hat installation into a powerful, low-cost testing environment. freeVSD works by simulating up to 250 full-featured private servers. Hard links to system files facilitate compact and homogeneous environments for each virtual server. Logins to the virtual servers are restricted via the native chroot facility, effectively creating a secure sandbox.

Now, we can experiment recklessly, hand the keys over to inexperienced juniors or casually grant root privileges to strangers, with little concern for negative consequences.

From the system administrator's standpoint, freeVSD enables you to create multiple self-sufficient systems, each with its own administrative account and the ability to manage user accounts, as well as the ability to configure their own web services, mail services, database server—a “Lite” version of Linux, if you will.

freeVSD was originally developed for an ISP in the United Kingdom and has been under development for three years. Based on mailing list archives, freeVSD seems popular and well supported. Questions are answered quickly, either by users or the developers.

Many significant functions of each virtual server can be administered by a rootlike account named Admin. For example, the Admin account can add users, manipulate their privileges, make changes to httpd.conf, restart various aspects of the server and so on.

Installing freeVSD can be a bit tricky. You need to be especially careful if you intend to restore the hostsystem back to its original configuration. As always, it is imperative to back up anything you are not comfortable with catastrophically losing. According to the web site, support for Debian, Mandrake and Slackware is forthcoming, but so far only Red Hat 6.x and 7.x. are officially supported. Version 1.4.6 introduces support for Red Hat 7.0, but Red Hat 6.2 seems to have more of the kinks worked out.

It is recommended that freeVSD be installed on a nearly pristine system. Start with a freshly installed Red Hat 6.2. Then decide whether you want any special server software available, such as MySQL, Postgres or PHP. Apply patches. Ideally, all applications should be installed before configuring freeVSD. Note that freeVSD works quite well under VMware, which might prevent a bit of stress during the first few installs. You'll probably need around 800MB of free disk space to accommodate the filesystem skeleton.

I assume you have or can obtain a FQDN or dedicated IP number for your first virtual host (freeVSD uses IP aliases). Of course, you need to be sure to obtain permission from whomever is in charge of your network before engaging in behavior that might be considered aggressive.

Then choose a name for your first virtual host. A good idea might be the hostname, (e.g., “myhost” if your FQDN is myhost.mydomain.com) or the domain name (mydomain), if you are providing hosting for multiple domains.

Here's an overview of the freeVSD install process, described in detail by the file /usr/doc/freevsd-x.y.z/user-guide.txt.

At this point, assuming nothing went wrong, you will have a functioning virtual server to which you may connect via Telnet or FTP.

To uninstall, stop all virtual servers with /usr/sbin/vsboot --stop. Next, optionally delete existing virtual hosts with

/usr/sbin/vsdadm vs_delete localhost myhost

Then, run /usr/sbin/vsd-uninstall.pl to restore configurations and optionally delete files. Take care to answer these questions correctly the first time as you won't get a second chance, and you will have to restore configurations manually. Finally, remove the pkgs and main RPMs.

The /usr/sbin/vsd-genskel.pl installation script copies system files from the host into a directory referred to as the skeleton. This process is controlled by entries in /etc/freevsd.conf, which specifies files to include and delete. The copies are placed in /home/vsd/skel/ by default:

$ ls /home/vsd/skel
bin  dev etc  home  lib  proc sbin  tmp  usr

The root filesystem of each virtual server relies on hard links to corresponding binaries in this skeleton. A hard link is an additional directory entry pointing to a particular file. Note that hard links differ from symbolic links in that every hard link must be deleted before the file is removed from the system; if the file that a symbolic link points to is removed, the symlinked file becomes unresolvable. Since each server shares the same copies of files that comprise most of the default filesystem, relying on hard links creates a tremendous savings of disk usage.

As far as ps and top are concerned, the freeVSD user processes seem to run as root; however, they don't start with root privileges. By default, UIDS start at 1,000 and increment by 200 for each subsequent virtual machine (i.e., the first vm starts at 1,000, the next at 1,200, etc.). This behavior can be modified by changes to the configuration file, /etc/vsd.conf.

As mentioned earlier, the host machine assumes multiple IP addresses using IP aliasing. A dæmon, /usr/sbin/virtuald, works with inetd (or xinetd, which replaces inetd in Red Hat 7.0) to intercept client connections to services such as Telnet or FTP. The incoming connection is handed to the appropriate dæmon in the virtual environment by using chroot to the host's filesystem, by default a directory located in /home/vsd/vs/.

Because of potential security exploits, the virtual web server does not run directly on port 80. Instead, a host server process called vsredirect forwards traffic from port 80 to port 8080 and moves https traffic on port 443 to port 8443. The file documentation file security.txt details how a malicious user could gain root access without this safeguard in place. Redirection is recommended for all privileged ports below 1000.

Within the freeVSD filesystem, several common commands such as rm, ls and passwd have been modified slightly so that the Admin account has the additional privileges required to administer accounts on the virtual server. This includes the ability to establish user accounts and manage their files.

Separate dæmon processes (httpd, pro-ftpd, sendmail and so on) are created for each virtual host. A suid script allows the Admin of the virtual host to start and stop the dæmons via the /usr/sbin/rebootvs command.

The Admin account has simulated root privileges, enabling the Admin user to perform various administrative tasks without seriously compromising the host. The Admin is not a diluted root account; rather, it is an enhanced user account with the limited ability to manage files and accounts on a particular virtual server.

Upon logging in, whoami reports admin and your home directory is /root. Examination of the root (/) directory reveals that Admin owns /root, /home and /tmp. Other Admin-owned files can be determined by running

find / -name admin -print

You are really running Linux, and it really is the bash shell. It's not a watered-down experience. You can run Python or Perl or even compile new apps with gcc. You are in a standard Linux shell environment, and it appears to be your own system.

In /home/httpd/docs one can find the DocumentRoot for the default web server, and /home/web/log contains the log files. The httpd.conf file is located in /etc/httpd/conf/ and may be modified by Admin. As mentioned earlier, /usr/sbin/rebootvs will effectively restart the virtual server by restarting its processes.

Examining /etc/passwd shows the regular system accounts and the admin account; note that this file is read-only. However, /usr/sbin/useradd <newuser> works as expected, including the addition of the new user to the /etc/passwd file.

The file /etc/vsd/priv has a format similar to /etc/groups. It controls which users have privileges such as login, Telnet and FTP access, as well as whether they are allowed to run Perl or gcc. The /usr/bin/listrights command will show the rights granted to a named user. The /usr/sbin/setrights command is a utility to manage this file; however, from a review of the source code (setrights.c), it doesn't appear that the login privilege may be granted with this utility. It is possible to edit /etc/vsd/priv manually and grant this privilege; since Admin doesn't have write privileges in /etc/, the login privilege can only be granted by root.

The freeVSD dæmon follows the Sysvinit startup conventions and can be controlled via:

/etc/rc.d/init.d/vsd start
/etc/rc.d/init.d/vsd stop
/etc/rc.d/init.d/vsd restart
/etc/rc.d/init.d/vsd status

(though this may not yield accurate results).

Several options are configurable through the file /etc/vsd.conf. You may also start or restart a given virtual server with the /usr/sbin/vsboot command.

The /usr/sbin/vsdadm command, introduced during the installation process, can be used to create and delete virtual machines and to manage user accounts. It works by issuing commands to the vsd dæmon that runs on port 1725. This approach enables alternative front ends to manage freeVSD. This includes the administration suite, which consist of tools developed by Idaya, Ltd., some of which run in a web browser or under Microsoft Windows.

The program /usr/bin/bevs (become a virtual server) allows a user on the host to quickly assume the role of root on the virtual machine, without Telneting or otherwise connecting to the virtual machine. Most notably useful is the ability to set the Admin password initially or manage files not owned by Admin or a virtual user.

The following scripts, while useful during install, may become liabilities afterward. You may wish to issue the command chmod a-x on them to avoid inadvertently executing them again:

/usr/sbin/vsd-genskel.pl
/usr/sbin/vsd-install.pl

The program /usr/sbin/vsd-refreshskel.pl will update the freeVSD skeleton and then relink files in the root filesystems on each virtual machine. This enables you to add, remove or update applications available to the virtual servers. Be sure to read the documentation or scan the source code before trying this command. Packages you wish to make available to the virtuals, but not to the host, may be installed with the command

rpm -ivh --force --root=/home/vsd/skel/ file.rpm

rpm -ivh --force --root=/home/vsd/vs/some-vs file.rpm

Since /usr/sbin/vsd-refreshskel.pl will refresh the skeleton, it's useful to propagate upgrades to system software on every virtual machine. Please note that this utility seems to be broken for Red Hat 7.x in freeVSD version 1.4.6.

The /usr/share/doc/freevsd-1.4.6/ directory contains documentation and is made available to each virtual machine by default. Finally, /usr/share/freevsd/ contains scripts useful for site customization.

Giving superuser capabilities to regular users is serious business. We frequently see root exploits via shell accounts, so one must wonder if these problems could be compounded in the chroot environment provided by freeVSD.

On the other hand, one operating system patch can instantly improve security for up to 250 web site administrators. Further, having dedicated httpd processes prevents many of the problems usually present in a typical shell environment, where one user's actions could inadvertently disrupt another user's service.

An ordinary user on the host can see the processes of every user on every virtual machine. Further, using the bevs -r command, they can become root on any of the virtual servers by default! The bevs command is suid root, so apparently this is intentional. I recommend changing the permissions in order to disable this (e.g., chmod o-xr /usr/bin/bevs will do the trick), while still enabling members of the root group to run bevs.

The virtual server administrative dæmon /usr/sbin/vsd listens on port 1725. Out of the box, it is a trivial task to query a virtual server for user lists, change user privileges—in effect, fully control the vsd dæmon remotely, with no authentication. As suggested in the file security.txt, it is imperative that you control this with commands on the host, such as these (which should be placed in rc.local):

ipchains -A input -p tcp -s W.X.Y.Z
<@cont_arrow>å<@$P>--dport 1725 -j ACCEPT
ipchains -A input -p tcp -s 0/0 --dport 1725 -j REJECT

Because of the complexity of propagating operating system updates to the virtual machines, it is prudent to run a stable distribution.

Perhaps the simplest way to test out a new application is to keep a spare machine around on which you can safely format the drive and re-install it. While reinstallation can be a good experience builder, continually installing and tweaking becomes a nuisance.

Another option is to create disk images. After you have freshly installed the operating system and have it configured to your satisfaction, a command such as

dd if=/dev/hda1 of=/tmp/image

can create an image file that can be used to restore the operating system at a later date. Beware of the 2GB file size limitation if using this strategy—the split utility will be helpful.

The commercial application, VMware, fully simulates an x86 PC, right down to a full-featured Phoenix BIOS. For maximum performance, VMware directly utilizes the host processor to execute most instructions, instead of emulating the CPU. The VMware approach is not particularly efficient from a disk utilization standpoint: each session must have the operating system completely installed. VMware can create virtual disks from files, so multiple drives are not necessary. In addition to being disk-intensive, memory must be dedicated to each running virtual. However, with VMware, you are not limited to a single Linux distribution or even to the Linux operating system. It offers a more complete virtual server environment but is more resource-intensive, including the need to administer and update each operating system separately. Note that Plex86 (http://www.plex86.org/) is a free alternative to VMware under development.

User Mode Linux offers yet another means. In short, the Linux kernel has been ported to another architecture—a process running inside a regular Linux kernel. Full details are available at http://user-mode-linux.sourceforge.net/.

freeVSD is a promising product. The technique of running multiple server processes for each virtual server results in a robust and reasonably efficient solution. The illusion that the virtual server is a dedicated machine is fairly convincing, but the lack of full control over the virtual's filesystem can be occasionally frustrating. Plus the Admin user's inability to install RPMs is somewhat limiting. But for moderate development, especially where the primary tools you need are already installed and compatible with freeVSD, this is one of the quickest ways to create a fresh Linux system. The chances of inadvertently goofing up the host from a virtual server are negligible.