Managing your Logs with Chklogs

One of the attributes that characterizes Unix systems, and therefore Linux, is true multitasking. Because of it, there are usually many processes running on your machine, including the not-so-evil daemons and other important programs such as uucico (Unix-to-Unix copy-in copy-out). If your system is properly configured, those programs leave traces of their activities in the system logs. These logs usually contain lots of data that must be filtered to generate more readable reports. In the case of system problems, these logs are a valuable source of information for tracking, and possibly solving, the problem.

Years ago, when I was faced with the prospect of spending some of my free time looking through raw logs and trimming them so that they wouldn't eat up disk space, I decided it was time to take action—I wrote the program Chklogs. ChkLogs is an acronym for “Check Logs” in the Unix tradition.

Whether you are an experienced Linux user, system administrator or a newcomer to the Linux world, you will certainly find this subject of interest. Although it is mainly a system administration tool, it also has applications in the user world.

In this article I will introduce you to version 2.0 of Chklogs which should be out by the time this is published. Currently version 1.9 release/build 2 (1.9-2) is available, which is the last Perl 4.0x compatible version. Version 2.0 and higher require Perl 5.003—which is not much to ask considering about 99% of registered users have that Perl version.

Here are some of the features offered by Chklogs:

Logical Log Groups are groups of logs that have something in common; this could be a collection of UUCP logs, INN logs or anything you consider a group. You can use virtually any name; the only restriction is that the name must be valid to create a directory. Chklogs has two built-in groups: syslog and common (reserved).

Alternate Repository (AR) is a directory where the archived logs are stored after they have been processed if they met the threshold condition. By default, archived logs are created with tar and GNU zip. A local AR is a directory named OldLogs under the directory in which the log resides. A global AR is a directory hierarchy (you define the root of this AR) where log archives are divided into logical groups. Here, you will always see the “common” subdirectory where all “orphan” logs go, i.e., those that have not been explicitly declared to belong to a group.

Log Shuffling is well known and is also called log rotation. The “phased-out” log is assigned a number or tag each time until a maximum is reached, at which point the oldest one is removed.

Directory Lumping is another nice feature. Instead of specifying each log separately by name, you supply a directory name. Chklogs treats any non-archived, non-directory file (as recognized by Chklogs) as a log and acts on it according to the action specification. A real-world application of this option is a site that gives UUCP access to a large number of subdomains (very cumbersome to do by hand), and a separate UUCP transfer log is kept for each site. Chklogs always determines first if you have specified a directory or a file.

Every time I make the first release/build of a new version (e.g., v2.0-1), I submit it to Sunsite and Funet FTP sites. Whenever a fix is required (and therefore no new features), I put out a new release/build (e.g., 2.0-2) only on the primary site, my ISP. To make sure that you have the latest version, check out the primary site and/or the Chklogs web page at http://www.iaehv.nl/users/grimaldo/info/.

Now, get “version 2.0 build 1” installed on your system. Unpack it and go to the root of the directory tree:

gunzip chklogs-2.0-1.tar.gz
tar xvf chklogs-2.0-1.tar
cd chklogs-2.0-1

Under the root tree is the bin directory containing the scripts and modules which comprise the Chklogs package. The /doc directory contains all the necessary documentation, including man pages in both troff and HTML format. A plug-out directory contains some extra utilities to scan your logs—use them as is or as examples to build your own. Last is the /contrib directory. In the root of the tree there is the README file, release notes, a makefile for installation and, most important, the GuideMe script. Type:

GuideMe

Assuming you have made the necessary configuration parameter changes (mailer, compress program, administrative account, library location, Perl location etc.), you are now ready to actually make Chklogs work for your system. I will also assume it has been installed (see Makefile) in /usr/local/sbin and /usr/local/lib/Degt/ by the make command (make install). The same library directory is shared by the graphical user interface.

If you are not yet sure you wish to commit your logs to Chklogs, you should make a back-up copy of them in a local directory. Run Chklogs on those copies until you feel safe, and you will. Doing this requires use of the resource file (see below).

The Chklogs package consists of an administrative program (chklogsadm), the main program (chklogs) and some Perl modules (Smtp.pm, Chklogs.pm, Interp.pm). For the second part of the configuration, we need the Chklogs configuration file (chklogs.conf) and, optionally, the resource file.

The resource file (.chklogsrc) resides in your home directory. You need it if you wish to first test on a copy of your logs or if you don't have root permissions (therefore, not used for the system logs but for others). When Chklogs runs, it first gets configuration information that is coded in the scripts configuration variables (the configuration section is clearly marked in the source), then it looks for a Personal Resource File (PRF) and then the command-line options, in that order.

In the PRF you can override some variables from the script configuration, for example, the admin user and others. Here is an example provided in the distribution:

# Personal resource file for Chklogs 2.x
set ChklogsConf ~/devel/test/chklogs.conf
set ChklogsDb ~grimaldo/devel/test/.chklogsdb
set VarRun  ~/devel/test
set RelativePath MyOldLogs
set Admin  grimaldo
mode ignore on
set SyslogConf /etc/syslog.conf

You will notice it has a syntax similar to Tcl. That is, with the exception of the mode lines, everything between ignore on and ignore off is not interpreted. The Tcl/Tk GUI does have a dummy mode routine, but all variable settings are performed as this ignore mode is only meaningful in the Perl mode. Personally, I find the dollar sign in front of normal variables annoying. This same resource file (with some extensions) is used by the GUI. The PRF example from above does not show the variables that are used exclusively by the GUI. However, for the examples in this article, I will assume you are working on the actual logs and not a copy.

In the above script, all variables are overrides for the internal configuration variables. ChklogsConf indicates the location of the configuration file (introduced in the next section), ChklogsDb is the internal database (you must never edit or move this file around), and VarRun is the directory where the PID lock files are kept (including where to find Syslog's and Chklogs' PID file). RelativePath is used when you choose the local repository, and Admin is the e-mail address where the reports are sent when the mail command-line option is specified. Normally, I don't need this file when running Chklogs, but I do use it during testing as a non-privileged user.

Listing 1 is a sample configuration file demonstrating the various features. This file contains three types of lines: comments/spacers, directives and specifications. It is divided into two major sections, the header and the body:

The header is comprised of the comment lines that identify this configuration for a particular host, and the two directive lines which must appear in the order shown. The header can be created with either of the following two administrative commands:

chklogsadm newconf -global /var/log/Chklogs

or

chklogsadm newconf -local

In the instruction section or body, we find mostly specification lines. These lines follow a particular syntax:

LogName Threshold Action [Parameter(s)]

The first field is a fully-qualified log name, the second is a threshold that can be a size or age. The size must be in bytes or, optionally, append the K qualifier for kilobytes. If you opt for an age, it must be in days by appending a D (e.g., 7d) or in months by appending an M (e.g., 2m). The qualifiers D, M and K are case insensitive. The log is always compared to the threshold, if it is reached then the specified “Action” is taken.

The Action field can have a value of archive, execute or truncate, which has no arguments, Chklogs uses the truncate system call to chop the file to a length of zero. Use this option if you don't care about this log or after you have used the log with either the archive or execute options as shown in Listing 1.

The archive action takes one parameter that indicates how many logs to keep archived before rotating/shuffling. If no parameter is given, an internal one is used (see documentation). Use this option, if you plan to look at this log at a later time.

The execute action is used to spawn a plug-out, that is, an external program that will operate on the log. This is useful if you have a particular log scanner that will filter the log and give you a human-readable report. You can specify as many parameters as you wish, the special parameter %L is replaced by Chklogs with the full name of the log and is used to inform the plug-out which file to process.

Listing 1 also shows a Logical Group definition. There, a group named NetNews is created so that, if a global alternate repository is used, there will be a NetNews directory under it in addition to the /common directory. For each group, you can associate an external program to execute in the plug-out interface. Parameters are allowed, but %L is meaningless in that context. Pre and Post specify which program to execute before and after the group has been processed respectively. A group definition always starts with the three directive lines in that order, then one or several logs and, finally, an empty line or a comment line. Do not include any of these in between the lines of the group definition.

In the case of the sample NetNews group, we want the daemon to stop using the logs before we touch them and restart (or continue) after we are finished. This is particularly useful for INN (which has a program to control the daemon) and HTTP. Chklogs handles syslog automatically; it knows what to do and how to do it.

Finally, notice that the last specification line refers to a directory as explained earlier in the Directory Lumping section.

Once you have created your first configuration file, you need to have Chklogs initialize its internal database (not to be confused with configuration) specified by either the configuration variable ResrcFile (for historic reasons) or the Personal Resource File's ChklogsDb variable. Remember, initialization overwrites any previous history so should be done only after the first installation (rather than when you make changes to add/remove logs/groups). Use the following administrative command:

chklogsadm init

Then, issue the administrative command:

chklogsadm initrepos

Finally, the sync option of chklogsadm is needed whenever you make a modification to the configuration file (chklogs.conf). This is not done automatically because it is not very effective if several modifications are made to the file consecutively. This operation is as simple as typing:

chklogsadm sync

Chklogs comes with several plug-outs. I use them to generate statistics on my UUCP logs, listserver log, etc. You can build your own plug-outs, and in fact I would like to hear about any log scanners or filters you have. Available plug-outs are stored in the /plug-out and /contrib directories.

A plug-out is any external program executed by Chklogs in the Pre/Post group directives or the execute actions. This program must not generate any output on stdout/stderr as it will clutter the report, and if you, like most users, run Chklogs with a cron job, then this sort of behavior is undesirable. So, do you have a nice scanner that does write to stdout/stderr and don't want to hack it up? Or don't know a thing about programming? Simply use the cdkwrap wrapper provided in the distribution, and it will capture all the output and mail it to you.

Also, the plugged-out child inherits certain environment variables that provide useful information:

Now, we have finished our setup; we have a configuration file, a resource file and initialized repositories. The administrative tasks are done for now (you can make changes later, if needed), and we are ready to get down to the action. Chklogs has several command-line options, some of which can be combined to achieve a particular effect. I won't cover all of them, or even all of the possibilities, but I will discuss enough of them to enable you to begin, and to acquaint you with a few of its capabilities. Note that we will now be using the chklogs program, not the administrative program, chklogsadm.

Basically, there are four major actions you can perform:

To check the correctness of the configuration file (although the check is not very thorough) use the following command:

chklogs -w

For a quick overview of which logs are archived into the repositories, there is the -l command line option:

chklogs -l

To get the usual log report indicating whether a log is still within its threshold, and if not, what action would be performed, use the -w option:

chklogs -w [-m]

Alternatively, you can also specify the -m (mail) option to mail the report. You cannot put them on the same switch (-wm).

Finally, when you want Chklogs to actually process your logs as specified in the configuration file, simply use it without any of the above options:

chklogs [-m]

A report is produced on standard output unless you use the mail (-m) option. Mail is the most frequently used option.

Most users make an entry into the /etc/crontab file so that Chklogs runs every day at a particular time and mails the report. A typical crontab entry looks like this:

# System Cron Tab
45 23 * * * root /usr/local/sbin/chklogs -m

I hope that you, too, will find Chklogs to be a useful package and will soon incorporate it into your system administration tool box. It is used by the Linux community and by most of the other major Unices (I counted 9 different flavors in my last scan of the registration database), for small home systems, service providers and other major network providers. I am always available by e-mail for suggestions, and if there is a problem I always give an answer. A standard form for reporting problems is provided with the distribution to ease my task. I would like to thank all of those who have sent suggestions and encouraging e-mail, as well as problem reports. It has been a great experience to provide this free software to the community. Chklogs is “postcard-ware”; I collect them.