Detecting Suspect Traffic

With the release of the much-awaited Linux 2.4.0 kernel early this year, GNU/Linux received a major boost toward entering the realm of enterprise-class operating systems. Several areas of the kernel have been improved since the 2.2.x series, not the least of which is the firewall code. Netfilter in the 2.4.x kernel series replaces the old ipchains firewall code in the 2.2.x series and incorporates many features that are required in an enterprise-class firewall, such as statefullness, DoS protection/rate limiting, network address translation (NAT), MAC address filtering and, last but not least, TCP packet filtering and logging based on arbitrary combinations of TCP flags. By contrast, ipchains (in addition to not being stateful, etc.) have the lamentable limitation of being able to distinguish between only two types of TCP packets: those that set and those that do not set the SYN flag. Netfilter's ability to distinguish arbitrary TCP flag combinations allows advanced port scans, such as those that are easily leveraged against a machine via nmap, to be detected. First we need a bit of background on nmap so that we can illustrate both how advanced port scans work and how to detect them.

nmap is the world's best-known port scanner and incorporates many advanced port scanning techniques that can be used to detect open ports on a target machine and/or network. In addition, nmap incorporates operating system fingerprinting and TCP sequence number prediction (among other things) to give better reconnaissance information than can be gleaned by determining which ports are open on a target machine. Three particularly interesting types of TCP scanning modes that nmap supports are FIN, NULL and XMAS scans. In normal TCP traffic, a FIN packet (or packet that has the FIN flag set) can be sent by either end of a TCP connection indicating that the session is finished or, more precisely, that there is no more data to be sent. A FIN scan works on the principle that an orphaned FIN packet (i.e., one that is not already part of an existing TCP session) sent to an open TCP port should not be acknowledged. If such a packet is sent to a closed port, however, a normal unfiltered TCP stack is supposed to respond with an RST packet. Hence, to scan a host with FIN packets, nmap simply sends a single FIN packet to each of the target ports and waits for RST packets to trickle back. Ports that do not respond with an RST are open (or filtered by a firewall). Both the NULL and XMAS scans function similarly, but instead of setting only the FIN flag, the XMAS scan also sets the URG and PSH flags, whereas the NULL scan utilizes TCP packets that have no flag set whatsoever.

The basic principle in setting up a secure firewall is to have a default deny stance in which all traffic not explicitly allowed by the firewall is denied or rejected. In addition, the firewall should be configured to log any unauthorized packets to a log file for analysis by the system administrator or by automated logfile watchers such as psad.

For more information on getting iptables to run on your system, I highly recommend Rusty Russell's “Netfilter HOWTO”, available at netfilter.samba.org.

# iptables -L
Chain INPUT (policy ACCEPT)
target  prot source   destination
ACCEPT  all  anywhere anywhere state RELATED,ESTABLISHED
ACCEPT  TCP  anywhere anywhere TCP dpt:ssh flags:FIN, SYN,RST,PSH,ACK,URG/SYN
ACCEPT  TCP  anywhere anywhere TCP dpt:www flags:FIN, SYN,RST,PSH,ACK,URG/SYN
LOG     TCP  anywhere anywhere LOG level warning prefix `DENY '
DROP    TCP  anywhere anywhere

Does such an iptables policy stops port scans? First, any TCP packet (regardless of TCP flags) sent to a destination port other than 80 or 22 will be logged by rule number four and then dropped by rule number five. This takes care of any noisy scans of any type against any of the 65535 TCP ports other than 80 and 20. What about scans that test whether 80 and 22 are open? The answer is that it depends upon the type of scan. Rules three and four accept packets that have the SYN flag set and all other flags cleared, so any scan that makes use of just SYN packets will succeed. Also, any scans that make use of normal TCP connect() system calls, as a normal web browser or SSH client would use, will succeed since rule number one allows three-way TCP handshakes to complete.

nmap supports both of these scanning techniques with the -sS (half-open or SYN scan) and -sT (TCP connect() scan) command-line switches. Any other scanning technique that is not based on legitimate TCP traffic to these ports will be blocked and logged. Such scans include the FIN, XMAS and NULL scans mentioned in the nmap section, as well as SYN/FIN and ACK scans.

Now that we are reasonably confident that our iptables firewall will block port scans, we should not become complacent with our success. It is not enough just to block scans; we should also do our level best to detect them, as a port scan could be a precursor to a more advanced attack.

The Port Scan Attack Detector (psad) is a program written entirely in Perl and designed to work with restrictive ipchains/iptables rulesets in order to detect both TCP and UDP port scans. It features a set of highly configurable danger thresholds (with sensible defaults provided), e-mail alerting, optional automatic blocking of offending IP addresses via dynamic reconfiguration of ipchains/iptables firewall rulesets, and verbose alert messages that include the source, destination, scanned port range, begin and end times, dns and whois lookups. For iptables firewalls, psad makes use of the extended logging capability to detect highly suspect scans, such as FIN, XMAS and NULL, that are easily leveraged against a target machine via nmap. In addition, psad incorporates many of the TCP and UDP signatures included with the Snort intrusion detection system to detect scans and/or traffic to various backdoor programs (e.g., EvilFTP, GirlFriend, SubSeven) and DDoS tools (mstream, shaft). psad is free software released under the GNU Public License and is available from www.cipherdyne.com.

The basic task of psad is to make use of firewall log messages generated by either ipchains or iptables to detect suspect network traffic. To accomplish this task, psad needs a way of efficiently getting the data it needs from the log messages the firewall writes to syslog. Hence, upon installation psad creates a named pipe called psadfifo in the /var/log/ directory and reconfigures syslogd to write all kern.info messages to the pipe. In syslog parlance both ipchains and iptables log messages are reported via the kern facility at a log level of info. The bulk of the work done by psad is accomplished by two separate dæmons: kmsgsd and psad.

The kmsgsd dæmon is relatively simple and its sole responsibilities are to open the psadfifo pipe, read any kern.info messages that indicate ipchains/iptables has denied or rejected a packet and write all such messages to the psad data file /var/log/psad/fwdata. Since the regex included within kmsgsd looks only for packets that have been denied or rejected, the fwdata file provides psad with a distilled data stream that contains information that may be significant exclusively from a security perspective. However, this information is only as complete and verbose as the firewall is able to generate, hence the need for the most restrictive possible firewall ruleset.

iptables does not support a logging option for any rule that drops or rejects packets. An easy solution to this, though, is to precede any drop rule with a logging rule that uses the --log-prefix option. See rule number four of simplefirewall.sh. The kmsgsd code snippet in Listing 1 illustrates reading firewall messages from the psadfifo named pipe and using the regex to match any dropped packet messages generated by either ipchains or iptables.

Listing 1. Reading Firewall Messages with kmsgsd

Once the fwdata file is populated with packets that have been denied, it is the responsibility of the psad dæmon to analyze and make judgments about whether or not the packets constitute a port scan or other suspect network traffic. psad accomplishes this by periodically checking fwdata for new lines indicating that packets have been denied recently by the firewall. Port scans are assigned a danger level from one to five based on how many packets are denied within a fixed period of time. However, a scan may also be assigned a danger level depending on whether it matches any of the set of nasty signatures contained within the psad_signatures file. Examples of such signatures include “NMAP Fingerprint attempt” (for any packet that sets the URG, PSH, SYN and FIN flags) and “DDoS - mstream client to handler” (for a SYN packet destined for port 15104).

After a port scan reaches a predetermined danger threshold, an e-mail is sent that contains several pieces of information: the source IP from which the scan originated, the destination IP, the range of newly scanned ports (TCP or UDP) within the last scan interval, the complete range of scanned ports since the beginning of the scan (TCP and UDP), the start and end times, the danger level assigned by psad, reverse dns information, the TCP flags that were used in the scan (along with the corresponding nmap command-line options that would generate such as scan) and whois information.

Instead of appealing to the whois client installed by default on any Linux distribution, psad makes use of an excellent whois client written by Marco d'Itri. This client has the advantage of accurately querying the correct whois database for almost any scanning source IP address. psad also features the ability to reconfigure an ipchains/iptables ruleset to block any IP address that has reached a sufficiently high danger threshold. This feature is disabled by default, since many administrators do not want volunteer administrators across the Net having the ability to force the firewall to block access to any web site they choose by spoofing nasty packets from the web site's IP address.

Some administrators, however, like the ability to set a high threshold for a port scan or other nasty traffic and have psad block it automatically. Note that any traffic analyzed by psad has already been blocked by the firewall. But if a scan reaches a sufficiently high threshold, the auto-blocking feature will block all traffic from the offending source IP, since the administrator would probably not want such an IP sending any packets to the local network, legitimate or not.

psad also includes a signal handler such that if a USR1 signal is sent to the psad process, psad calls Data::Dumper to dump the contents of the %Scan hash to the /var/log/psad/scan_hash.$$ file, where the $$ represents the pid of the psad process. The %Scan hash is the main psad data structure and contains all scan information, and hence dumping its contents is useful both as a way to capture a record of scan data and as a debugging tool for extending psad's feature set. psad is a work-in-progress with much on the to-do list, but it is being developed actively and also has a relatively short version-release cycle.

To illustrate the differences in the output of ipchains and iptables firewalls, we first compare log entries generated by an nmap XMAS scan.

The ipchains messages in Listing 2 were generated by an nmap XMAS scan of TCP ports 79 through 81. Recall that an XMAS scan sets the FIN, URG and PSH flags. First the nmap command and output is listed followed by the corresponding ipchains output. Note that ipchains makes no mention of which TCP flags were set.

Listing 2. nmap Command and Output with ipchains Output

Listing 3. iptables Messages from TCP Session with Web Server

Now we perform the same nmap scan (the nmap command line and output is identical to the above ipchains example, so it is not repeated) and display the corresponding iptables output (see Listing 3). This time we can plainly see the FIN, URG and PSH flags set in the packets used in the scan.

Resources