Security Monitoring and Enforcement with Cfengine 3

Aleksey Tsalolikhin

Issue #205, May 2011

Cfengine is well known as a powerful system configuration management tool, but did you know you also can use it to secure your systems?

Cfengine, from the start, has had security as a key part of its design and use scenarios. Here, I demonstrate how Cfengine 3 can be used to increase the security of a Linux system by monitoring file checksums, monitoring filesystems for suspicious filenames, monitoring running processes, monitoring open ports and managing sshd.conf.

Because Cfengine 3 is under active development, I suggest you install the latest version from the Cfengine Source Archive (see Resources).

The purpose of this article is to give practical examples of how you can use Cfengine to increase security on a Linux system. See the Quick Start Guide in the Resources section of this article for help in learning the Cfengine language. (I don't provide a tutorial on the Cfengine language here.) This article is based on Cfengine version 3.1.5a1.

Monitoring File Checksums

Cfengine 3.1.4 shipped with 214 unit tests that can double as examples of Cfengine's functionality. They are installed to /usr/local/share/doc/cfengine/. I've adopted unit_change_detect.cf into detect_changes_in_etc.cf (Listing 1).

Listing 1. detect_changes_in_etc.cf

# GNU GPL 3

###################################################
#
# Change detect
#
###################################################

body common control

{
bundlesequence  => { "detect_changes_in_etc"  };
}

###################################################

bundle agent detect_changes_in_etc

{
files:

  "/etc"

   changes      => detect_all_change,
   depth_search => recurse("inf");
}

###################################################

body changes detect_all_change

{
report_changes => "all";
update_hashes  => "true";
}

###################################################

body depth_search recurse(d)

{
depth        => "$(d)";
}

Run this with:

cf-agent -KIf detect_changes_in_etc.cf

cf-agent is the component of Cfengine that actually makes changes to the system. (There are other components to serve files, monitor system activity and so on. cf-agent is the piece that makes changes to the system, and the one you'd use to start learning Cfengine.) In the command above:

-K — tells cf-agent to ignore time-based locks and allows you to run cf-agent repeatedly (no “cool-off” period, which might otherwise kick in to prevent system overload).
-I — tells cf-agent to inform you of its actions and any changes made to the system.
-f — specifies the policy filename.

On the first pass, cf-agent builds a file information database containing file timestamps and inode numbers and builds an MD5 hash for each file. You should see something like this:

# cf-agent -KIf detect_changes_in_etc.cf
  !! File /etc/hosts.allow was not in MD5
     database - new file found 
  I: Made in version 'not specified' of
     'detect_changes_in_etc.cf' near line 22
  ...
#

There are two messages here, alert and info.

Cfengine prefixes its output to help you understand what kind of output it is (in other words, metadata):

Informational messages start with “I”.
Reports start with “R”.
Alerts start with !! or ALERT.
Notice of changes to the system starts with ->.

In the above case, the alert message is accompanied with an info message about the policy that was in effect when the alert was produced, its version number (if supplied) and the line number.

I didn't specify the version number, but the line number is useful. Line 22 is:

changes      => detect_all_change,

This is the line responsible for Cfengine adding /etc/passwd to the MD5 database. It tells Cfengine what to do about changes—to detect them.

Now, I run cf-agent again, and it runs quietly. The contents of /etc match the MD5 sum database:

# cf-agent -KIf detect_changes_in_etc.cf
#

Next, I edit /etc/hosts.allow to add “sshd: ALL” to simulate an unauthorized change. Watch cf-agent scream:

# cf-agent -KIf detect_changes_in_etc.cf
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ALERT: Hash (MD5) for /etc/hosts.allow changed!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 -> Updating hash for /etc/hosts.allow to
MD5=2637c1edeb55081b330a1829b4b98c45
I: Made in version 'not specified' of
'./detect_changes_in_etc.cf' near line 22
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ALERT: inode for /etc/hosts.allow changed
38901878 -> 38901854
ALERT: Last modified time for /etc/hosts.allow
changed Sat Jan 29 17:09:26
2011 -> Mon Jan 31 08:00:02 2011
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#

There are three alerts:

MD5 hash changed (because the contents changed).
The inode number changed (when vi saved the file).
The modification time changed (when vi saved the file).

Reminder: messages about actions that Cfengine takes are prefixed with “->”:

 -> Updating hash for /etc/hosts.allow to
MD5=2637c1edeb55081b330a1829b4b98c45

You can set up Cfengine to complain via e-mail or syslog, so even if the intruder tampers with the MD5 database, the alarm will sound. In commercial versions of Cfengine (Nova), you can set up multiple Cfengine nodes to share their MD5 databases and monitor and cross-check each other.

You can run this check fairly often—every five minutes, if you like and if your hardware will sustain it. (Computing lots of MD5 sums can be expensive on CPU and disk I/O.) Is the added security worth it to you?

Monitoring for Suspicious Filenames

Cfengine has a special cf-agent control variable called suspiciousnames. You can put a list of names into it to warn about during any file search (such as was done during the MD5 hash check). If Cfengine sees these names during recursive (depth) file searches, it will warn about them. If suspiciousnames is not set, cf-agent won't check for them. It's not set by default.

Let me demonstrate how this works by adding the following control block to detect_changes_in_etc.cf:

body agent control
{
suspiciousnames => { ".mo", "lrk3", "rootkit" };
}

A cf-agent control block controls the behavior of cf-agent. This is where you can set things like dry-run mode (don't change anything but report only on what changes would have been made—useful for learning Cfengine), the largest file size Cfengine will edit and so on. So the suspiciousnames variable is set in the agent control block. It's an array of strings.

Let's create a suspiciously named file to see cf-agent get excited:

# date > /etc/rootkit
# cf-agent -IKf detect_changes_in_etc.cf
Suspicious file rootkit found in /etc
#

So, if you're scanning your system directories for an MD5 hash check, you can add the suspicious name check too.

Monitoring Running Processes

I follow the best practice of securing servers by disabling unnecessary services. I often want to make sure my Web servers are not running CUPS—usually, a Web server does not need to print!

The example shown in Listing 2 is based on the Cfengine unit test unit_process_kill.cf.

Listing 2. cups_not_running.cf

body common control

{
bundlesequence  => { "cups_not_running"  };
}


########################################


bundle agent cups_not_running {


    processes:

            "cupsd"  signals => { "term", "kill" };

}

The line of interest in Listing 2 is:

processes: "cupsd"  signals => { "term", "kill" };

This means if there is an entry in the process table matching “cupsd”, that process will be sent TERM and then KILL signals:

# cf-agent -IKf cups_not_running.cf
 -> Signalled 'term' (15) to observed process match '28140'
 -> Signalled 'kill' (9) to observed process match '28140'
#

But, let's not be so brutal. Cfengine can report suspicious process names. You can keep an eye out for password sniffers, crackers, IRC bots and so on with the policy shown in Listing 3.

Listing 3. report_suspicious_process_names.cf

body common control

{
bundlesequence  =>
            { "report_suspicious_process_names"  };
}

###################################################

bundle agent report_suspicious_process_names

{

vars:

  "suspicious_process_names" slist =>
      {
          "sniff",
          "eggdrop",
          "r00t",
          "^\./",
          "john",
          "crack"
      };


processes:

 ".*"

    process_select  =>
      proc_finder("$(suspicious_process_names)");
}


###################################################

body process_select proc_finder(pattern)

{
     command => ".*$(pattern).*";

     process_result => "command";
}

The key line here is:

vars: "suspicious_process_names" slist => { "sniff",
    "eggdrop", "r00t", "^\./", "john", "crack" };

A variable called “suspicious_process_names” is a list of strings; what we deem as suspicious process names includes, let's say, any processes starting with ./. As you can see, this list can include regular expressions. Cfengine uses Perl-compatible regular expressions.

You can set the contents of this array to reflect what you consider suspicious process names. Then, Cfengine scans the entire process table (that's the processes: .*) and loops over the contents of the “suspicious_process_names” array. Cfengine has implicit looping over arrays, so if you have an array @{suspicious_process_names} and you reference ${suspicious_process_names}, you're actually saying:

for ${suspicious_process_names} in (@{suspicious_process_names}
do
  ...
done

That's what happens when you say process_select => proc_finder("$(suspicious_process_names)"); You're actually saying, for each element in @(suspicious_process_names), find processes that match that regex.

Anyway, I want this to be a security demonstration rather than a language primer, so let's continue:

# cf-agent -IKf report_suspicious_process_names.cf
 !! Matched: root     20044 20002 20044  0.0  0.0  
    4956  19   664    1 22:05 00:00:00 ./eggdrop 
#

The first numeric field (20044) is the PID. The last field is the process name. (Why is there an IRC bot on my Web server?)

Case Study

In 2000, David Ressman and John Valdes of University of Chicago reported in a LISA paper “Use of Cfengine for Automated, Multi-Platform Software and Patch Distribution” how they detected a cracker using similar functionality in Cfengine 2:

Since the people who break into our systems almost exclusively use the compromised systems to run sniffers, IRC bots, or DoS tools, we decided to make up a list of suspicious process names to have Cfengine look for and warn us about every time it ran. Besides the usual suspects (more than one running copy of inetd, anything with “sniff”, “r00t”, “eggdrop”, etc., in the process name, password crackers, etc.), we had Cfengine watch for any process with “./” in the process name.
One afternoon, we got an e-mail from Cfengine on one of our computers that had noticed that the regular user of that machine was running a program as “./irc”. It wasn't uncommon to see our users using “./” to run programs, nor do we have objections to our users running IRC, but in this case, it was a bit unusual for this particular user to be running an irc process (good UNIX system administration practice also dictates that you know your users).
Poking around the system, we discovered that the person running this program was not the regular user of the machine, but was someone who had evidently sniffed our user's password from somewhere else and remotely logged in to his system just minutes before Cfengine had alerted us. This person was in the process of setting up an IRC bot and had not yet tried to get a root shell.

You can add to your defense-in-depth by monitoring for suspicious process names.

Monitoring Open Ports

You can increase your security situational awareness by knowing on what ports your server is listening. Intruders may install an FTP server to host warez or install an IRC server for bot command and control. Either way, your server's TCP profile has changed (increased) in terms of on what TCP ports it listens.

By constantly comparing desired and actual open TCP ports, Cfengine quickly can detect an intrusion. Cfengine 3 runs every five minutes by default, so it can detect a compromise pretty fast.

The code example shown in Listing 4 starts with hard-coded lists of what TCP ports and corresponding process names are expected on the system: 22 sshd 80 httpd 443 httpd 5308 cf-server. It then uses lsof to get the actual list of TCP ports and process names, compare them and report DANGER if the comparison fails.

Listing 4. check_listening_ports.cf

body common control

{
bundlesequence  => { "check_listening_ports"  };
inputs  => { "Cfengine_stdlib.cf"  };
}

bundle agent check_listening_ports
{
vars:
"listening_ports_and_processes_ideal_scene"
string =>
"22 sshd 80 httpd 443 httpd 5308 cf-server"; 
# this is our expected configuration

vars:
"listening_ports_and_processes" string =>
execresult("/usr/sbin/lsof -i -n -P | \
/bin/grep LISTEN | \
/bin/sed -e 's#*:##' | \
/bin/grep -v 127.0.0.1 | \
/bin/grep -v ::1 | \
/bin/awk '{print $8,$1}' | \
/bin/sort | \
/usr/bin/uniq | \
/bin/sort -n | \
/usr/bin/xargs echo", "useshell"); # actual config.
# tell Cfengine to use a shell with "useshell"
# to do a command pipeline.

classes:
"reality_does_not_match_ideal_scene" not =>
  regcmp (
      "$(listening_ports_and_processes)",
      "$(listening_ports_and_processes_ideal_scene)"
  );  # check whether expected config matches actual

reports:
  reality_does_not_match_ideal_scene::
"
DANGER!
DANGER! Expected open ports and processes:
DANGER! $(listening_ports_and_processes_ideal_scene)
DANGER!
DANGER! Actual open ports and processes:
DANGER! $(listening_ports_and_processes)
DANGER!
";  # and yell loudly if it does not match. 
    # Note:  A "commands" promise could be used in
    # addition to "reports" to send a text message
    # to a sysadmin cell phone or to feed 
    # CRITICAL status to a monitoring system.
}

Here's an example run:

# cf-agent -IKf ./check_listening_ports.cf
R:
DANGER!
DANGER! Expected open ports and processes:
DANGER! 22 sshd 80 httpd 443 httpd 5308 cf-server
DANGER!
DANGER! Actual open ports and processes:
DANGER! 22 sshd 80 httpd 443 httpd 3306 mysqld 5308 cf-server
DANGER!!!
#

Again, this is a security demonstration, not a language primer, but if you want to understand the policy, follow the Quick Start Guide for Cfengine. If you need any help understanding this policy, come to the help-cfengine mailing list or ask me directly at aleksey@verticalsysadmin.com.

Managing sshd.conf

The next example is Diego Zamboni's Cfengine bundle for editing the sshd configuration file and restarting sshd if any changes were made. It has two parts (to abstract the under-the-hood details). In the first part, the sysadmin edits the sshd array to set variables corresponding to the sshd configuration parameters. For example, to mandate Protocol 2 of SSH, set:

"sshd[Protocol]" string => "2";

If the parameter is commented out, Cfengine uncomments it and sets it to the desired value. If the parameter is absent, Cfengine adds it and sets it to the desired value. Additionally, if any changes were made to sshd_config, sshd restarts to activate the change.

Listing 5. use_edit_sshd.cf

bundle agent configfiles
{
vars:
  "sshdconfig" string => "/etc/ssh/sshd_config";

  # SSHD configuration to set
  "sshd[Protocol]" string => "2";
  "sshd[X11Forwarding]" string => "yes";
  "sshd[UseDNS]" string => "no";

methods:
  "sshd" usebundle => edit_sshd("$(sshdconfig)", "configfiles.sshd");
}

Listing 6. edit_sshd.cf

# Parameters are:
# file: file to edit
# params: an array indexed by parameter name, containing 
# the corresponding values. For example:
# "sshd[Protocol]" string => "2";
# "sshd[X11Forwarding]" string => "yes";
# "sshd[UseDNS]" string => "no";
# Diego Zamboni, November 2010
bundle agent edit_sshd(file,params)
{
files:
  "$(file)"
  handle => "edit_sshd",
  comment => "Set desired sshd_config parameters",
  edit_line => set_config_values("$(params)"),
  classes => if_repaired("restart_sshd");

# set_config_values is a bundle Diego wrote based on 
# set_variable_values from Cfengine_stdlib.cf.

commands:
  restart_sshd.!no_restarts::
    "/etc/init.d/sshd restart"
    handle => "sshd_restart",
    comment => "Restart sshd if the configuration file was modified";
}

bundle edit_line set_config_values(v)

 # Sets the RHS of configuration items in the file of the form
 # LHS RHS
 # If the line is commented out with #, it gets uncommented first.
 # Adds a new line if none exists.
 # The argument is an associative array containing v[LHS]="rhs"
 
 # Based on set_variable_values from Cfengine_stdlib.cf, modified to
 # use whitespace as separator, and to handle commented-out lines.
 
{
vars:
  "index" slist => getindices("$(v)");

  # Be careful if the index string contains funny chars
  "cindex[$(index)]" string => canonify("$(index)");

field_edits:

  # If the line is there, but commented out, first uncomment it
  "#+$(index)\s+.*"
     edit_field => col("\s+","1","$(index)","set");

  # match a line starting like the key something
  "$(index)\s+.*"
     edit_field => col("\s+","2","$($(v)[$(index)])","set"),
        classes => if_ok("not_$(cindex[$(index)])");

insert_lines:

  "$(index) $($(v)[$(index)])",
      ifvarclass => "!not_$(cindex[$(index)])";
}

For an example of changes made, run diff of sshd_config before and after Cfengine edited it to set Protocol, X11Forwarding and UseDNS:


# diff /etc/ssh/sshd_config /etc/ssh/sshd_config.cf-before-edit
14c14
< #Protocol 2,1
---
> Protocol 2
95,96c95,96
< #X11Forwarding no
< X11Forwarding no
---
> X11Forwarding yes
> X11Forwarding yes
109c109
< #UseDNS yes
---
> UseDNS no
#

You may notice X11Forwarding is there twice after the edit, because it was in the file twice before the edit, once commented and once uncommented. But, this does not break things. Having X11Forwarding yes is valid syntax, and the /usr/sbin/sshd -t syntax checker does not complain.

You also may notice that cf-agent saved a copy of the original file, just in case.

Learning More

Download the source and follow the Recommended Reading on the Quick Start Guide site. Also, please visit us on the help-cfengine mailing list to share your ideas on automating security with Cfengine.

Resources

Cfengine Source Archive: www.cfengine.org/pages/source_code

Quick Start Guide: www.cfengine.org/pages/getting_started

“Automating Security with GNU Cfengine”, Kirk Bauer, February 5, 2004 (although based on Cfengine 2, the article gives an excellent overview of Cfengine's philosophy and power): www.linuxjournal.com/article/6848

Diego Zamboni's Cfengine Bundle for Editing the sshd Configuration File and Restarting sshd If Needed: https://gist.github.com/714948

Download the Cfengine Policies Used in This Article: www.verticalsysadmin.com/cfengine/LJ-May-2011