Anything that you can do from a shell command, you can do remotely with SSH. Here's how to set up keys for effective, secure remote tasks from cron jobs and scripts.
Using insecure protocols leaves your data and connected machines vulnerable to attack. Remote server management requirements demand that security be given a top priority. This article explains my process for using OpenSSH in unattended scripts and cron jobs.
Most readers are familiar with the secure shell (SSH) protocol, which creates a secure tunnel for commands, data and passwords to travel across the network. Recently, my workplace was faced with some challenges in securely setting up scripts through cron. We use SSH because it resolves the major problems with rsh; rsh sends clear text over the network and has weak host-based authentication. But our challenge became how to deal with password prompts when using SSH in unattended jobs. For example, we run df -k, top and swapon -s to get remote server statistics and alert our team if problems exist.
If you also still are using rsh, the SSH client, ssh, makes a perfect replacement in scripts. Modifications typically are minor. For example:
for host in $servers do rsh $host df -k done
simply becomes:
for host in $servers do ssh $host df -k done
SSH supports a wide range of authentication systems, the most common being Kerberos, Rhosts, Public-key and Password. Because we didn't have a Kerberos infrastructure in place, our readily available options to solve this problem were to echo the password in the script, use Rhosts, use ssh-agent or use public keys.
Our first options presented some challenges and weaknesses. First, simply echoing the password in a script is not a simple task, as SSH does not read from standard input. To make it do so would require advanced scripting techniques. More important, you would need to put your password either in the script itself or in a file on the filesystem. Although you could set proper permissions, getting access to the password would be a relatively easy task for a determined intruder. It could be as simple as restoring data from a backup or even viewing the password on-screen. This method was not an option for us.
Second, we considered host-based authentication, which is how users executing the rsh command are granted access. Because users are granted or denied access based on the host they are logging in from, no password is needed. This solution may work in some situations where security concerns are light, but the ability to pretend to be another host, to IP spoof and to disrupt DNS does exist. Also, due to the fact that once a host has been impersonated successfully, all users have been compromised on the remote host, we decided we needed something more secure.
Our third option was to use ssh-agent. Before we discuss this option here, though, we need to cover public keys and their use. Instead of using a plain-text password, SSH has the ability to use public key cryptography. This means that when a client connects to a server, it has a conversation with the server and proves its identity based on advanced mathematical computations.
Let's walk through the setup to generate a set of public and private SSH keys to allow a user named scripts to log in from hostA to hostB, assuming the user exists on both hosts:
1) Generate the keys:
[scripts@hostA]$ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/scripts/.ssh/id_dsa): Created directory '/home/scripts/.ssh'. Enter passphrase (empty for no passphrase): XXXX Enter same passphrase again: XXXX Your identification has been saved in /home/scripts/.ssh/id_dsa. Your public key has been saved in /home/scripts/.ssh/id_dsa.pub. The key fingerprint is: 41:03:aa:dc:cc:b9:39:50:65:bc:ee:7b:36:d2:64:7a scripts@hostA
2) Copy public key to hostB from hostA:
scp /home/scripts/.ssh/id_dsa.pub \ hostB:/home/scripts/.ssh/authorized_keys
scp is an encrypted replacement for rcp and simply copies files in a secured manner.
The authorized_keys file is a file that contains the public identities, or public keys, of users who can log in to your account by using public key authentication. All users maintain their own authorized_keys file, which typically lives in the hidden .ssh directory in a user's home directory. Users also may configure security restrictions to public keys here as well, which we review below.
The authorized_keys file is not created when you first run ssh-keygen to create your public and private keys. As a best practice, we recommend permissions of 600 for this file.
At this point, userA should be able to log in to hostB without a password using public key technology. Now, of course, we still have the same problem with echoing the passphrase into a script. As I mentioned, SSH does not take input from standard input, so this represents the same scripting challenge as before. To eliminate the need to retype your password continually, SSH comes with ssh-agent. You use ssh-agent as follows, in combination with ssh-add:
[scripts@hostA scripts]$ ssh-agent bash [scripts@hostA .ssh]$ ssh-add id_dsa Identity added: id_dsa (id_dsa)
We pass our shell to ssh-agent, and it inherits the keys we add with ssh-add. Now we need only type our passphrase once and we can use our key default key, id_dsa.pub, to be authenticated. An important note about using multiple keys in an interactive session with SSH is how you need to call SSH. For example, if you have created three private keys—your default key, id_dsa, and two other keys called backup and monitor to use for different tasks—you simply would call SSH with the -i parameter. This is done to make sure you're using your new key while logging in to the remote SSH server:
[scripts@hostA]$ssh -i backupkey hostB
When you are using ssh-agent, you may believe you simply type in ssh -i backup to use your backup identity. This is not quite the case, though, as the ssh-agent typically uses the key that is on the top of its key list. To get a listing of all the keys you have loaded in ssh-agent, run ssh-add -l for a listing of fingerprints of all identities currently loaded in the agent:
[scripts@hostA scripts]$ssh-add -l 1024 df:ab:8e:d7:e4:bd:35:f6:b3:2e:76:6b:dd:71:2f:fe monitor (DSA) 1024 4e:4c:00:ba:1e:5d:60:08:f2:b8:2e:d4:59:1e:ff:2f id_dsa (DSA) 1024 0a:72:24:9e:0a:cd:e2:e4:5f:93:cb:ac:66:78:03:f6 backup (DSA)
Because ssh-agent typically favors the key listed first, it favors the monitor key. To be able to use the backup key, you need to unset the shell variable SSH_AUTH_SOCK and then point SSH to the identity you want to use, as follows:
[scripts@hostA scripts]env -u SSH_AUTH_SOCK \ ssh -i backup hostB
After doing this, you will be using the proper key as intended.
Using ssh-agent is, of course, a great time saver for interactive use. When used in scripts, however, a human still needs to type in the passphrase at least once when the machine boots. This ends up being the best we can achieve with ssh-agent, even with scripts to automate most of the procedure. For more information on that topic, refer to SSH, The Secure Shell: The Definitive Guide. Ultimately, the ssh-agent option also did not meet our needs in deploying secure batch jobs, as our goal was to automate the jobs totally.
That left us with the option of using public keys without a password. The remainder of this article focuses on that setup, how to secure it further and some options to consider when using this setup. In any environment, thorough planning and review of security polices should occur before deploying or modifying security configurations.
The first method in securing our setup is to use the from= directive in the authorized_keys file. The syntax looks like this:
from="host1,host2" KEY
What this says is allow only users from host1 or host2 and authenticate them against the public key matching KEY. For example, to restrict logins from only hostA and hostB for our user scripts, the authorized_keys would look like this:
from="hostA,hostB" ssh-dss AAAAB..Aqbcw= scripts@hostA
This is by no means a foolproof restriction. As I mentioned before, it is possible to pretend to be another host and spoof an IP. But this restriction adds a layer of security and increases the effort needed to compromise our host. Notice that I intentionally shortened the key, which is quite long, due to space constraints. Be aware that the from= syntax is sensitive to short and long DNS names. HostA is not the same as HostA.somewhere.
Our second line of defense in securing our script setup is to use the command ="" directive, also specified in the authorized_keys file. The syntax for this looks like:
command ="command", KEY
This tells SSH to run command and then exit. It effectively limits your ability to run commands on the remote server. As you might expect, you can combine both of these in your authorized_keys file; simply make sure you separate the options by a comma:
from="hostA,hostB",command="/bin/df -k" ssh-dss AAAAB3N...Aqbcw= userA@hostB
Now, should someone compromise this user and key, the worst that can be done is retrieving a listing of disk space on the remote host. In fact, this is the only command you can run with this key. In order to run multiple commands securely, you have a few options. First, consider calling a script instead of command. For example, run top, df -k and hostname from a shell script named myscript.sh and set command="/path/to/myscript.sh". Second, if you need to run multiple commands at different times during the day to same host, you could create another key for your user. This time, use the -f option to specify a file other than the default:
[scripts@hostA]$ssh-keygen -t dsa -f backupkey Generating public/private dsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in backupkey. Your public key has been saved in backupkey.pub. The key fingerprint is: 14:ac:c5:5f:65:69:2f:8d:cf:0a:70:9e:5c:4e:c7:84 scripts@hostA
You would copy the contents of the new public key, backupkey.pub, to the authorized_keys file of the users on the hosts you want them to access, the same as the default key. Be sure to set the new command="" for your new key and the new command you want to run.
Now, you would use the -i parameter to make sure you're using your new key while logging in to the remote SSH server:
[scripts@hostA]$ssh -i backupkey hostB
Finally, you could create a separate user for this task. For example, you could create one user to monitor disks and one to automate backups. Each configuration has its advantages and drawbacks. The question to answer here is, “Do you want to manage keys or user accounts?” I prefer to have different keys and make a note of them with a comment.
One piece of the SSH key we have not considered yet is the comment field. The default comment for userA's public key created on HOSTA is userA@HOSTA. Basically, everything after the key is ignored as a comment. So to keep track of keys and what they are used for, I make a comment in the remote server's authorized_keys file. For example:
ssh-dss AAAAB3NzaC1kc3MAAA......Jw= scripts@hostC-Key used for disk monitoring
Our third line of defense is the ability to limit the traffic we forward. We have three main options here that merit discussion. First, the no-port-forwarding option means what is says. When this key logs on, the ability to forward TCP/IP ports is denied. Forwarding ports is a great way to bypass firewalls; hence, the account used to run scripts should be given the ability to run only the commands necessary. The ability to forward TCP/IP ports is a potential security problem, so we restrict it.
Second, no-X11-forwarding tells the SSH process not to forward any X11 connections upon login. Any attempt to do so returns an error. We see that this is simply another way for an intruder to exploit our hosts, so we disable it. Again, we try to lock down what the account that logs in can do, but we also permit it to perform its job.
Third, no-agent-forwarding in the authorized_keys denies this key the ability to forward its ssh-agent and stored keys to another host. This reduces complexity and also takes away another avenue for a potential intruder to trespass.
The final option in the authorized_keys file we want to use is the no-pty option, which says not to allocate a pseudo-tty when logging in. Non-interactive commands continue to work using the associated key; however, you can no longer issue commands through an interactive session. Should an intruder gain access to your private key and somehow circumvent the other options, this option effectively ensures that he or she cannot issue interactive commands to do any damage.
With the above options in place, we have a reasonably restricted key that still can perform its job. Our final authorized_keys file looks like this:
from="hostA,hostB",command="/bin/myscript.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-dss AAAAB3....o9M9qz4xqGCqGXoJw= scripts@hostA
Before we finish our discussion on options, let's look at two more that are not related directly to security. When running SSH in scripts, we use the -q and -o “BatchMode=yes” command-line options. The -q stands for quiet mode. The man page for sshd sums this up nicely: “Nothing is sent to the system log. Normally the beginning, authentication, and termination of each connection is logged.” This is useful for suppressing warnings otherwise interpreted as command output. The -o “BatchMode yes” makes sure SSH does not prompt the user. So our script changes a little more:
for host in $servers do ssh -q -o "BatchMode=yes" $host df -k done
Because we are specifying an option on the command line, we are certain the options will not be overridden as they take precedence. Typically, the global client configs are looked at first, usually /etc/ssh_config; then the local client configs, usually ~/.ssh/config; and finally the command line. As several versions and variations of SSH are available, always consult the man page for correct locations and syntax.
Ensuring proper options are set for each particular key and using a layered security approach goes a long way in making your servers less vulnerable to attacks. Setting the least privileges possible reduces the potential damage done during a successful attack. Using these methods, your data and networks become more secure and still run efficiently.
Resources for this article: /article/8400.