Getting Started with Salt Stack—the Other Configuration Management System Built with Python

I was proudly wearing one of my Salt Stack shirts the other day when my daughter asked me, “What is Salt Stack?” I began by explaining the problem it solved. If you have multiple servers and want to do things to those servers, you would need to log in to each one and do those things one at a time on each one. They could be fairly simple tasks like restarting them or checking how long they have been running. Or, you might want to do more complicated things like installing software and then configuring that software based upon your own specific criteria. You also might want to add users and configure permissions for them.

What if you have ten or maybe even 100 servers though? Imagine logging in one at a time to each server individually, issuing the same commands on those 100 machines and then editing the configuration files on all 100 machines? What a pain! Just updating user password policies would take days, and introducing an error would be quite likely. What if you could update all your servers at once just by typing one single command? The solution? Salt Stack!

Like my daughter, you may not have heard of Salt Stack (saltstack.org), but you might be familiar with Puppet (puppetlabs.com) and Chef (opscode.com). Salt is a similar tool, but it's written in Python, is relatively lightweight as far as resources and requirements, and it's much easier to use (in my opinion). Salt uses the 0MQ (www.zeromq.org) communication layer, which makes it really fast. It also is entirely open source, licensed under the Apache2 (www.apache.org/licenses/LICENSE-2.0) license, and boasts a vibrant and productive community.

There currently aren't any plans to release a crippled community version or a more feature-rich paid enterprise edition either. With Salt, the version you get is the version everyone else gets too—whether you've paid money or not. There are plans for an enterprise version, but it merely will be less bleeding-edge and will be subjected to a higher amount of testing and quality assurance, and it possibly will include training as well.

Tools like Salt, Puppet and Chef allow you to issue commands on multiple machines at once, and install and configure software too. Salt has two main aspects: configuration management and remote execution.

Salt Stack is a command-line tool. There isn't anything to click on with your mouse, and the feedback is presented as text that is returned on your screen. This is good. It keeps things lean, and most servers don't include a graphical user interface anyway. (Note: I use the terms Salt and Salt interchangeably throughout this article. They mean the same thing in this context.)

In this article, I cover the two tools included with Salt. The first is remote execution, although there isn't any clear delineation or any different way to interact with Salt if you want to work with configuration management or remote execution. This allows you to log in to a master machine and then execute commands on one or many other machines at once. With Salt, you simply type your command once on your master machine, and it executes on every machine, or even a targeted group of machines.

Second, Salt is capable of storing configuration directives, and then instructing other machines to follow those directives by doing things like installing software, making configuration changes to the software, and then reporting back on the progress and success or failures of the installation.

Later, I demonstrate using Salt to install an additional package on one, or even 1,000 machines, and then configure that package by issuing just one command.

Salt is a constantly evolving organism. Possibly by the time you read this, some things may have changed. You always can find the most current documentation here: docs.saltstack.org/en/latest/index.html.

You do need a few prerequisites before installing Salt:

Even though Salt is designed to interact with multiple servers, for this tutorial, you actually can accomplish everything on one machine.

Use your package manager to install Salt, and follow the installation guide found in the Salt Docs for your particular distribution (docs.saltstack.org/en/latest/topics/installation/index.html). You'll also need sudo or root privileges to use Salt and install these packages.

The benefits of using a package manager or installing from source are a constant source of on-line and water-cooler debates. Depending on your distribution, you may have to install the packages from source instead of using your package manager.

If you'd like to install from source, you can find the latest Salt source files in the Salt Project's GitHub repository (https://github.com/saltstack/salt).

After following the instructions for installing both a salt-master and salt-minion, hopefully, everything went well and you didn't receive any errors. If things didn't work out quite right, support is generally available quickly from the Salt Stack mailing list (saltstack.org/learn/#tab-mailinglist) and the #salt IRC channel.

The terms master and minion refer to the controller and the controlled. The master essentially is the central coordinator for all of the minions—similar to a client/server configuration where the master is the server, and the minion is the client.

For this tutorial, I cover issuing salt-master and salt-minion commands on the same machine. If you are configuring multiple machines, choose one to be the master, and all the others will be minions. The choice of master or minion is yours, and there are many reasons to configure one machine as the master. I explain how to set one as a master and the other(s) as minions next.

Salt's configuration files are located in /etc/salt. By default, these files are named minion and master. If you've installed the salt-master and salt-minion on the same machine, you will see two respective files, master and minion.

You first need to tell your minion how to locate and communicate with your master. Even though you are running both on the same server, you still need to tell your minion where your master is.

Now that your minion knows where your master is, it's time for them to authenticate one another. Salt uses public key encryption to secure the communication between master and minions. You need to notify the master and minion that they can trust each other by accepting the minion's keys on the master.

Accept your minion's keys using the salt-key command. Salt automatically takes care of generating these keys for you, so you simply need to accept the minion(s) you want.

Now that you have a salt-master and a salt-minion, and the minion and master trust one another, you can check the connection by issuing a test ping command from the master. This will return “True” if your master can communicate with your minion. Type salt '*' test.ping, and it should return:

>{1st-Salt-Minion: True}

Note that the wild-card '*' targets every minion, and as you have only one, this is basically moot (it's just faster than typing salt '1st-Salt-Minion' test.ping).

If you receive a “True” response back from your minion, you have installed Salt Stack successfully and configured your master and minion to communicate properly.

If you don't, you may want to restart your master and minion without the -d (dæmon) flag, so you can observe the output. For more information, see the Salt documentation at docs.saltstack.org/en/latest/topics/configuration.html.

The Salt command syntax involves the command, the target(s) and the action. So, for this example, '*' targets everything (it's a wild card), and test.ping is the action.

You can now execute any available command on any connected and authenticated minion. Important note: these commands must be available on the targeted minion in order to execute them. For instance, a command like:

sudo salt '*' cmd.run "service apache2 restart"

would work only for a distribution that calls the Apache Web server apache2 and that has the Apache Web server installed. For others, you would need to issue the command:

sudo salt '*' cmd.run "service httpd restart"

Some other examples might include querying the amount of time your servers have been running. You can do that with:

sudo salt '*' cmd.run "uptime"

If you had, for example, Apache Bench installed on a master but not on a minion, the command:

sudo salt '*' cmd.run "ab -n 10 -c 2
 ↪http://www.google.com:80/index.html"

would fail if you tried to execute it on a minion, since Apache Bench isn't installed on the minion.

The possibilities here are practically limitless. You can reboot all of your machines at once, update system software and check your machines' health from one terminal instead of logging in to each machine and issuing these commands independently.

You also can target specific groups, based upon criteria that you select. See the -G flag documentation at saltstack.org for more options.

Very rarely should you ever need to log in to a minion again. All configuration and execution can be handled remotely, quickly and simultaneously.

Now that you've installed Salt and can execute remote commands, why stop there? The second part of Salt's power comes from the configuration management tools included with Salt.

If you haven't used any type of configuration management system before, here is a simple example. Say you have a set of configurations and packages that you generally install for every Web server. You can keep these configuration directives in small text files and then instruct your servers to install these packages and configure them to your liking, every time you create a new server. You also can use configuration management to keep all of your servers updated once they have been created and respond to changes in packaging or new configurations.

Let's install the libpam-cracklib package, so you can add additional requirements for user passwords. I chose this package because it is useful for almost any server connected to the Internet. It allows you to set additional password requirements regarding length, and it requires that your users' passwords contain special characters or numerals. You easily could substitute any particular package you want. These examples do require that the package be available in your system's package manager though.

Salt's configuration management directives and files are, by default, kept within the /srv/salt directory. This is where all your configuration files and any files you want to copy to any of your minions reside. Salt also includes a file server system as part of the configuration management features. Salt doesn't touch your master's system files though, so don't worry; all configuration management takes place within the /srv/salt directory.

Salt, by default, uses PyYAML (pyyaml.org) syntax for its template files, but numerous other templating languages are available as well. Be sure to follow the proper formatting techniques for YAML, which involves two spaces instead of tabs. I have found the on-line YAML parser (yaml-online-parser.appspot.com) to be invaluable when troubleshooting syntax issues with YAML files.

To enable the configuration management functionality within Salt, you need to edit your master configuration file once again. In /etc/salt, open your master file and locate the lines that refer to file_roots. In the default configuration, this was around line 156. Now, uncomment this directive by removing the # from the following lines:

file_roots:
  base:
    - /srv/salt

This tells Salt where to locate your configuration management files. Depending on how you installed Salt, you may need to create the /srv/salt directory.

The base configuration file is known as a Top File, and it resides within the /srv/salt directory. Let's create one now. This file provides mappings for other files and can be used to set a base configuration for all servers. Again, with your favorite text editor, create a top.sls file within the /srv/salt directory. You can think of this file as a roadmap for different directions for each minion. Within your top.sls file, add the following lines:

base:
  '*'
    - servers

The base directive lets Salt know that this configuration is a base configuration and can be applied to all machines. The wild-card '*' targets every machine. The - servers directive is an arbitrary name that allows you to recognize what the directive pertains to. Feel free to choose something that makes sense to you. This entry also refers to a particular configuration file that you will now create to install the libpam-cracklib.

After you save your top.sls file, create a new file called servers.sls within the /srv/salt directory. This file will hold your specific configuration, including the name of the package to be installed and also a reference to a configuration file. In the new servers.sls file, add the following:

libpam-cracklib:
  pkg:
    - installed

The first line is the name of the package specifically how your package manager refers to it. For example, the Apache HTTP server is called apache2 in aptitude-based package manager distributions, but httpd in yum-based package management systems. Make sure you use the proper name for the package depending on which package manager you are using. You can target specific package names using what Salt refers to as grains. Refer to the documentation for more information and advanced examples of using grains in SLS files to target distribution-specific systems (salt.readthedocs.org/en/latest/topics/tutorials/states_pt3.html#using-grains-in-sls-modules).

Lines 2 and 3 tell Salt what to do with this package. For this example, you want it installed. To remove a package, you simply would change - installed to - removed. Remember, spacing is very important! On line two, there are two spaces before pkg:, and on the third line, there are four spaces before - installed. If you receive any errors, check your syntax via an on-line YAML parser.

In order to install the libpam-cracklib package, you need only the first three lines of this file. You could stop here, and libpam-cracklib would be installed with the default configuration supplied by your package manager. You then would need to log in to the machine on which it is installed and configure it for your particular needs. This defeats the purpose of using configuration management, and Salt offers a solution to this as well.

Salt can act as a secure file server and copy files to remote minions. In this same servers.sls file, add the following lines:

/etc/pam.d/common-password:
  file:
    - managed
    - source: salt://servers/common-password
    - require:
      - pkg: libpam-cracklib

Take note of line 4; this is where you tell Salt your particular file's location, and the lines after that tell Salt what package is required for this file. The line - source: salt:// maps to your /srv/salt directory on your master.

After you've saved your servers.sls file, make a new directory under /srv/salt called servers. This is where you will store your configuration file for the libpam-cracklib.

When you are installing packages and configuration files, you may want to install them first on a test server, and then configure them to your liking. Then you can copy the configuration files into your /srv/salt location. This way, you can verify that the configuration is functioning properly before deploying it to multiple servers.

Now your configuration will be available to Salt, and you can place this configuration on every minion, along with installing the libpam-cracklib package. Your /srv/salt directory should look something like this now:

/srv/salt
            top.sls
            servers.sls
          /servers
                  common-password

I'm using the libpam-cracklib here as an example, but this technique will work for any software that has configuration files associated with it. For instance, you easily could modify your Apache httpd.conf file to include your server's hostname and configure virtual hosts.

With all of your sls files in place and configuration files ready to go, the last step is to tell Salt to configure your machine remotely. The state.highstate command is what triggers this synchronization. Using the previous syntax to target all machines, enter this from the command line:

sudo salt '*' state.highstate

Hopefully, after a brief amount of time, your minion will return a success that looks something like this:

>>
  State: - pkg
  Name:      libpam-cracklib
  Function:  installed
      Result:    True
      Comment:   Package libpam-cracklib installed
      Changes:   wamerican: {'new': '7.1-1', 'old': ''}
                 cracklib-runtime: {'new': '2.8.18-3build1', 'old': ''}
                 libcrack2: {'new': '2.8.18-3build1', 'old': ''}
                 libpam-cracklib: {'new': '1.1.3-7ubuntu2', 'old': ''}
                   
----------
  State: - file
  Name:      /etc/pam.d/common-password
  Function:  managed
      Result:    True
      Comment:   File /etc/pam.d/common-password updated
      Changes:   diff: --- 
+++ 
@@ -22,7 +22,7 @@
 # pam-auth-update(8) for details.
 
 # here are the per-package modules (the "Primary" block)
-password   requisite   pam_cracklib.so retry=3 minlen=8 difok=3
+password   requisite   pam_cracklib.so retry=3 minlen=14 difok=3 
 ↪dcredit=1 ucredit=1 lcredit=1 ocredit=1
 password   [success=1 default=ignore]   pam_unix.so obscure use_authtok
 ↪try_first_pass sha512
 # here's the fallback if no module succeeds
 password   requisite   pam_deny.so

As you can see, Salt installed the libpam-cracklib package and then copied the common-password file from the master to the minion in the /etc/libpam-cracklib directory.

This was a fairly simple example on just one minion, but if you've ever had to install a LAMP-based Web server, imagine the amount of time you can save simply by using Salt's configuration management. Storing these settings in text files allows you to duplicate and create identical servers quickly.

You now have the ability to execute remote commands on multiple machines at once and store your configurations in easily maintained text files. You can install software packages specific to a type of server too.

With a little effort in the beginning, you can create one or many servers with your own specific configurations in the amount of time it takes for the packages to download to each machine. Salt doesn't execute these sequentially either. The commands are mostly implemented simultaneously on each machine, and if one minion happens to fail, the others will continue their progress.

Installing Salt can pay off big dividends later by allowing you to create specific-use servers based on a tested and repeatable configuration.

Visit the Salt Project page for more detail, and be sure to check the links for the mailing list, user-contributed documentation and examples. You'll find the community very welcoming and eager to lend assistance with any issues you encounter.