MILLE-XTERM provides a scalable infrastructure for massive X-terminal deployment.
Linux-based X terminals are well known for making computing affordable, for giving a second life to old hardware and for lightening administrative burdens. If you ever have toyed with the idea of using Linux X terminals, you are probably familiar with the Linux Terminal Server Project (LTSP), described in Figure 1. An LTSP server is perfectly suited for small workgroups or classrooms. However, in order to deploy a greater number of terminals, say thousands of them, the current LTSP model encounters scalability problems.
Figure 1. LTSP Simple Network Configuration
The main goals of the MILLE-XTERM Project are:
Unlimited scalability.
Centralized X-terminal management.
Enhanced user experience.
The MILLE-XTERM Project applied clustering concepts to the X-terminal infrastructure to achieve these goals.
The MILLE Project is funded by Canadian public agencies and school districts in the province of Quebec. MILLE means “Free Software Infrastructure Model for Education” and is targeted at educational institutions. It is composed of four subprojects: a portal (based on uportal), an open-source middleware stack, a CD with free software for Windows/Mac and, finally, MILLE-XTERM (the object of the present article).
The solution entails centralizing servers in a secure, air-conditioned computer room to form a cluster of terminal servers. The cluster has four major components, as shown in Figure 2. The first is the boot server, which provides DHCP and TFTP services and serves as a base system optimized for the terminals via NFS. Next comes the configurator, which generates the lts.cfg configuration file from an SQL database. The terminal then queries the load balancer, which in turn seeks out the cluster's least-loaded application server. The chosen application server login screen then appears, and after a successful authentication, the user can start using the desktop, browser, office suite and other applications.
Figure 2. Clustered LTSP Network Diagram
MILLE-XTERM relies on central file and authentication services that provide users with the same account and file on every application server. The open-source choice is NFS for users' home directories and OpenLDAP for the directory service. It also can be integrated into a Novell or Windows environment with additional configuration.
Unlike LTSP, there is no need for a separate network dedicated to terminals. They can share a LAN with other PCs. However, a reliable network infrastructure is crucial. With usual usage, each terminal generates an average of 1Mb/sec of X11 traffic. Low-end hubs should be avoided; managed switches with full-duplex capabilities really make a difference.
Unlike LTSP, each component is built from RPMs; system administrators easily can add features and local applications with standard package manager tools. The init scripts from the distribution are replaced by standard LTSP scripts. We are currently using Mandriva 2006 as the base distribution, though other distributions can be supported.
The boot server is mainly a read-only NFS server. Optionally, it can provide DHCP, TFTP and NBD swap services. The xtermroot contains a base system and an X server. With more than one boot server, it is easy to rsync the xtermroot periodically on each boot server. The terminals then boot in a uniform way, whichever boot server they use.
Diskless terminals need a way to store configuration data, such as screen resolution and available printers. Under LTSP, a central file called lts.conf stores the configuration of terminals and has to be edited manually. With thousands of terminals, you need a hierarchical database—that's the purpose of the configurator.
This component is written in PHP and has two interfaces. The first is dedicated to terminals. During the boot process, the terminal requests its configuration from the server using its MAC address as a parameter. The server generates the corresponding configuration and sends it to the terminal in the standard lts.conf format. A wrapper around the getltscfg command ensures backward compatibility with the other LTSP scripts.
The other interface lets administrators manage the configuration of the terminals via a Web browser. Administrators can organize terminals hierarchically by groups and apply configurations according to specific criteria, such as location or hardware type. But the configurator serves yet another function. It is designed to work with links, a console text browser, as shown in Figure 3. The terminal can boot in a special admin mode that does not require running the X server. To boot in this mode, the option mode=admin is appended to the kernel options in the bootloader configuration. Then, links is launched with the terminal configurator URL and MAC. The administrator can change the terminal settings directly. When complete, the terminal reboots and receives its new configuration.
Figure 3. Xterm-Based Administration System
The configurator also is useful for building terminal inventories. Hardware information is sent to the configurator during the boot process. Administrators can generate reports regarding the state of the terminals. Also, every connection to the configurator is logged and then can be analyzed to determine terminal usage, user login information and much more. You know how managers like reports!
When a terminal boots, it requests a display from the application server. To dispatch users on available application servers, MILLE-XTERM provides a load balancer. The first version of the load balancer (proof of concept) required five lines of PHP and returned a random address from a static list of application servers. Although simple, this approach had some drawbacks. First, an off-line server should be removed from the list and not be returned to the terminals. And, to provide reliable load balancing, several factors, such as number of processors, speed and load average have to be taken into account. Therefore, a much more robust and complete Python system has replaced the initial prototype (Figure 4). The load balancer agent runs on every application server, collecting data on the state of the application server and waiting for load-balancer server requests. The balancer is also a Python script that runs on the boot server. It contacts each load-balancer agent to determine its state and computes a weight for each server. A greater weight indicates that the server is less loaded and will be selected more often statistically to accept new users. A terminal request for an application server will then prompt the load-balancer server to get a randomly chosen application server in the weighted list.
Figure 4. Load Balancing the LTSP Cluster
Let's examine a concrete example: three application servers and two boot servers. Install the mille-xterm-lbagent package on each application server, and install mille-xterm-lbserver on each boot server. Make sure that the respective services are started, lbagent and lbserver. Add one node entry for each application server in the file /etc/mille-xterm/lbsconfig.xml:
<?xml version="1.0"?>
<lbsconfig>
<nodes>
<group default="true" name="PROD">
<node address="http://10.0.0.1:8001" name="xapp1"/>
<node address="http://10.0.0.2:8001" name="xapp2"/>
<node address="http://10.0.0.3:8001" name="xapp3"/>
</group>
</nodes>
Copy this file on every boot server. Fire up a browser and enter the URL of the load balancer to see it in action. By default, lbserver listens on port 8008, so don't forget to append the port to the URL: http://localhost:8008/. IP addresses of the chosen application server will be displayed. Press the refresh button to get a new IP and you're set!
The MILLE-XTERM Project focuses on user experience. A MILLE-XTERM X terminal has to provide the same functions as a regular workstation. In order to achieve this goal, extensive desktop personalization is required. Here are some of the most important topics:
Sound support: because the applications run on the server and the sound card is on the terminal, using /dev/dsp doesn't work. Solutions such as Esound, Arts, Nas and others have been developed over time. We have found that Arts can make a machine unstable (for example, TiMidity++ takes up to 99% of the CPU) and that very few applications support Nas. In our experience, the only workable solution is Esound (esd). GNOME support is native, and KDE, which uses Arts, has an option to use esd as its back end. Major applications, such as Flash Player, RealPlayer and SDL-based games, use it natively. When users first log in to the system, a script automatically creates configuration files to use esd. However, there is still work to do. For example, Audacity does not yet support esd.
Video support: with Windows, people usually use QuickTime, RealPlayer or Windows Media Player. They expect these essential packages to work on an X terminal. This is a bit of a challenge, as most distributions do not provide these players and appropriate codecs. Those familiar with the command-line swear by MPlayer, and others swear at it. A Mozilla plugin called mplayerplug-in solves the problem and neatly embeds MPlayer in the Web browser with Play, Pause and Forward buttons. MPlayer also can use Windows DLLs optionally to play popular file formats, but as with all proprietary programs, there are many licensing issues to consider.
Local applications: some applications simply cannot operate with the standard client/server scenario. For example, if users want to control the volume of the terminal sound card, they must run Aumix or Alsamixer locally on the terminal. Also, running the media player directly on the terminal is better than streaming 30 frames per second over the network, as this can quickly clog the network. In these instances, these applications run better when bunched locally on the terminal. To that end, a wrapper script called runlocal connects to the terminal with the user key via SSH and mounts the user's home directory via sshfs, allowing the application to access user files and settings.
Application configuration: most applications will create their own configuration files when first launched. In many cases, the default works well. In other cases, they need to be modified for use on an X terminal. For instance, several applications use the X-server memory as a cache memory. Although this is very efficient on a Linux workstation, it can cause an X-terminal crash when the memory used by the X server is bigger than the RAM of the terminal. An effective way to diagnose problems with X-server memory consumption is to use the xrestop tool. For KDE users, this kind of memory problem may occur when copying/pasting a large drawing with Klipper enabled. The only solution at this point is to disable Klipper. Here's a sample Firefox configuration that disables memory and disk cache:
// File /opt/firefox/greprefs/xterm.js
pref("general.config.obscure_value", 0);
pref("general.config.filename", "firefox.cfg");
// File /opt/firefox/firefox.cfg
lockPref("browser.cache.disk.enable", false);
lockPref("browser.cache.memory.enable", false);
lockPref("browser.sessionhistory.max_total_viewers", 0);
Personalization: many projects are focusing on language issues, such as internationalization (i18n) and localization (l10n). However, still no personalization layer (p13n) provides an easy way to configure icons on the desktop, menus, browser preferences, bookmarks, backgrounds, screensavers, default applications and so forth. Work has begun in the freedesktop.org project and with Sabayon and Kiosk, for instance, but it remains far from complete and covers only KDE and GNOME. In the meantime, we use a set of homemade scripts and config files to configure the desktop according to an organization's needs.
Global infrastructure integration: to be successful, the system must be integrated fully into the existing infrastructure (directory server, files and printing). Take printing, for example. If a school district has 50 schools and more than a thousand different printers, how do you select which ones to display in Firefox or OpenOffice.org? We solved the challenge with a simple wrapper that intercepts calls to the CUPS shared library in order to apply a filter based on the user location. The printer list and the default printer for the terminal is stored in the configurator database.
For security and administrative considerations, making isolated MILLE-XTERM components is possible. To gain the benefits of virtualization without performance drawbacks, Linux-VServer is the perfect alternative, although a few specific configurations are needed to install MILLE-XTERM inside a Linux-VServer. When installing a boot server in a vserver, it is not feasible to generate the initrd, unless the vserver has the CAP_SYS_ADMIN property set. The solution is to use a chroot on the host. Also, a user-space NFS server is used instead of the regular kernel-based nfsd. Finally, GDM on the application server will try to launch X inside a vserver, which is not needed. To correct this, append the --no-console option to the init script and it will listen only for network requests with no local host display.
When mastered, these few tricks allow you to add or remove application servers, copy existing application servers, back up and update them, and when satisfied with the changes, put them into production and duplicate them throughout the cluster, thereby elevating manageability to a higher level.
MILLE-XTERM can go further in a number of ways—beginning with security, or the lack thereof, as is the case of the XDMCP protocol. You can try it at home. Start an X session with Xnest and capture packets with ethereal. The following filter lets you view every keystroke typed:
x11.eventcode == 2
You could solve the problem with a local secure display manager that creates an SSH tunnel to encrypt the X11 traffic. Another possibility is to use OpenVPN between the terminal and the application server.
Almost every component of the MILLE-XTERM Project should be highly available. Work is in progress for the configurator (using slony replication for the PostgreSQL database). The boot servers (as well as the load balancer) will follow in order to have transparent failover (this can be achieved easily because their main functionality is as a read-only NFS server).
Optimizing the X protocol in order to save bandwidth is another interesting development. One can then use an X terminal with a simple broadband Internet connection. The next step for Linux terminals is NX/FreeNX. Last year, Linux Journal devoted five articles to the topic. NX clients would run locally on terminals, which would then require them to be added to the xtermroot in order to work.
Currently, more than 800 terminals are deployed with MILLE-XTERM at the Laval School District (one of the founders of the MILLE Project), and the plan is to deploy more than 1,000 additional terminals yearly (up to 75% of the existing computers will become X terminals).
We strongly believe that Linux terminals are the key solution that will allow school districts to provide a low-cost/high-quality desktop experience. With a cluster of Linux terminal servers, children can access the software they need to learn, create and be part of the Linux revolution.
The authors would like to thank the founders of the MILLE Project as well as the early adopters of the MILLE-XTERM solution:
Laval School District (www.cslaval.qc.ca): 800 terminals and still counting.
Mille Iles School District (www.cssmi.qc.ca): 300 terminals and still counting.
Grandes Seigneuries School District (www.csdgs.qc.ca): 100 terminals—pilot project.
Coeur des Vallées School District (www.cscv.qc.ca): 75 terminals—pilot project.
Affluents School District (www.csaffluents.qc.ca): pilot project.
Resources for this article: /article/9134.