Hack and /

When I first started out in systems administration, I had only a few machines to keep track of. It was relatively easy to remember which servers did which functions (mostly because one or two machines did just about everything). If a server had a problem, I immediately knew everything it would impact.

For better or worse, nowadays my position has become more complicated. When you personally manage tens or hundreds of machines, it can be difficult to keep everything straight. When a server goes down, you might no longer know what services are impacted or who else to notify. Beyond that, there's also the dreaded running-the-right-command-on-the-wrong-server mistake. I think every sysadmin has typed halt, rm -rf or some other destructive command in the wrong terminal at least once (just ask my old boss Bill).

In this column, I discuss some methods I've found to help you keep track of your servers. Although I can't guarantee you'll never type a command on the wrong server, I can say that as your environment grows to hundreds of servers, these techniques will help you pick up where your brain left off.

The message of the day (motd) is the message that greets you every time you log in to your system on the command line. For instance, here is the message of the day on one of my old Debian servers:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

The programs included with the Debian GNU/Linux system are free 
software; the exact distribution terms for each program are 
described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
No mail.

Messages like this are pretty generic, so it's easy to take them for granted and leave them alone. After all, in this example, I already know the OS, hostname and kernel version (Linux, napoleon, 2.6.20-1-k7). You can extend this information, however, and list anything you want.

The message of the day is managed in a file called /etc/motd. It's a simple text file, so you can modify it to say anything you want, although you'll want to limit it to what can fit on a standard console screen. Note that on modern Debian-based systems, the /etc/motd file is somewhat dynamic, so you will want to modify /etc/motd.tail instead.

So, how can you use this file to your advantage? A lot of security-minded administrators add a special terms of use in this file to note that their systems are private and do not allow unauthorized access. In that case, the motd acts like a No Trespassing sign, so if someone hacks in to the system, law enforcement has help demonstrating that the attacker was notified that it was a private system.

Although you may or may not want to add a No Trespassing sign to your motd, there are a number of other things you can add to the motd to make your life as an admin simpler. For instance, you could add a short set of documentation about the server, including what the server does, other groups to contact if there is a problem on the machine and even any special locations where custom files are stored. That way, when you log in, instead of a boring default motd, you could get something more like:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

Welcome to Napoleon. 
Local services: DNS, DHCP, Internal Wiki (http://wiki.example.net)

DNS config: /etc/bind, /var/named.
DHCP config: /etc/dhcpd.conf
Wiki files: /var/www/wiki

Support team: root@example.net, wikiadmin@example.net

You even might want to use the motd to pass along useful tips to regular users on the system. For instance, let's say your users use vim to view log files. On some systems, vim stores a complete copy of any files you open in /tmp. Although that's fine for a small text file, when you have users opening 1GB+ Apache logs, your /tmp space fills up quickly, and you are paged again and again. One solution might be to add a gentle reminder in your motd to use less, not vim, to read large text files.

Another great way to help remind you which servers you are on is to tweak your shell prompt. If you are a good security-minded admin and become root only when necessary, a quick tip is to make the root prompt a different color (like red), so it stands out and reminds you that everything you do is with root privileges.

There are many different tastes when it comes to a custom shell prompt, so you might want to tweak this to suit your preferences. Also, I'm assuming you will be using the bash shell that most systems tend to default to these days, so the file you should edit is /root/.bashrc. What shows up in your prompt is defined by the PS1 environment variable, so if you are curious what it is set to by default, simply type:

root@napoleon:~# echo $PS1
\u@\h:\w\$

In this example, you have a very basic prompt that lists the current user (\u), the @ symbol, the hostname (\h), a colon, the current working directory (\w) and a # symbol (if I'm root), or a $ otherwise (\$). On my sample system, it would look like root@napoleon:~# when I log in as root.

There are plenty of other ways you can tweak the prompt, and if you are curious, the full list of aliases you can use for it is found in the bash man page—just search for PS1.

Because I'm focused on colorizing the prompt and not necessarily changing the format, I mostly will leave the prompt as is. There are a few ways to colorize the prompt, but the simplest way I've found is to define some of the potential colors you'd like to use in environment variables ahead of time, and then you can assign them to the PS1 variable without going cross-eyed from all the escape characters. Open up /root/.bashrc, and if PS1 already is defined, add these lines above it:

NORMAL=`tput sgr0 2> /dev/null`
BOLD=`tput bold 2> /dev/null`
RED="\[\033[31m\]"
GREEN="\[\033[32m\]"
BLUE="\[\033[34m\]"
GREY="\[\033[1;30m\]"
PURPLE="\[\033[0;35m\]"

Now that all the colors are defined, I simply can define PS1 with the default settings, only with these color settings around it:

PS1 = "$RED\u@\h:\w\$$NORMAL"

Once you save the changes to .bashrc, the next time you log in, you will notice your prompt is colorized. Now you can spend the rest of the afternoon tweaking the prompt with different sets of colors and symbols like I did the first time I found out about it. It even might be worthwhile to use a different prompt color scheme for different types of servers.

One of the problems with the previous two methods is that you must log in to a machine to get information on it. That leads me to one of my favorite ways to organize my servers, DNS TXT records. Most people probably are familiar with a DNS A record (it maps a hostname to an IP address) and probably CNAME and PTR records (it maps one hostname to another hostname and an IP address to a hostname, respectively), but many admins aren't aware of (or don't use) TXT records. A TXT record essentially allows you to assign text to a particular hostname. If you have an internal DNS infrastructure for your machines, you probably already have A records for all your servers. If you add a TXT record as well, that gives you a nice centralized place to document what each server does in a way that can be queried from any machine on the network.

To demonstrate how to use TXT records, let's assume I'm using a standard BIND server for DNS, and this is a short section of the file that defines A records for three hosts—napoleon, snowball and major:

napoleon    IN   A   192.168.1.6
snowball    IN   A   192.168.1.7
major       IN   A   192.168.1.8

All I would do is add a new TXT record below any A records I have that lists what those servers do:

napoleon    IN   A   192.168.1.6
napoleon    IN   TXT "DNS, DHCP, Internal wiki"
snowball    IN   A   192.168.1.7
snowball    IN   TXT "Primary Internal File Server" 
major       IN   A   192.168.1.8
major       IN   TXT "Failover Internal File Server" 

Once I save my changes and reload BIND, the TXT records are ready to go. The next time I'm scratching my head trying to figure out what snowball does, I just have to issue a dig query:

$ dig snowball.example.net TXT +short
"Primary Internal File Server"

Note that I used the +short option with dig. That way, I get back only the contents of the TXT record instead of the volume of data dig normally gives me. Not only does this make it easy to narrow in on the information I want, it also makes it a handy little one-liner to add to other programs. I even could see some savvy administrators tweaking their shell prompt or motd so that it contained this value.

Again, the beauty of using TXT records to document this is that it puts the information in a central place that you control and that you typically have to modify whenever you add a host anyway. Just be careful if you use this for externally facing DNS hosts—you might not necessarily want to broadcast all of your server info to everyone on the Internet.