Insure++

For the last four years I have worked as a programmer writing software to find errors in integrated circuit designs. During this time I have learned a lot about chasing bugs. Ideally, you want to find and fix a program's bugs before you ship it to your customers. Remarkably, customers seem to be extremely creative people who can figure out how to use (and break) programs in ways programmers have never foreseen.

In order to combat the “creative user” problem, there is a type of program which will take source code or object files and produce a version that analyzes itself for bugs as it runs. The wonderful thing about this type of program is that it allows you to find bugs that are not causing any visible problems, so that you can fix them before they cause anyone trouble. We have found these programs to be invaluable at work.

Parasoft Corporation produces one of these programs, which they market under the name of Insure++. We recently evaluated the Solaris version of Insure++ at work, and I was excited to learn that they also had a Linux version.

A few weeks later I got e-mail from a sales person at Parasoft. She introduced herself and offered to put me in touch with their programmers if I had any technical problems. She went on to tell me that their product currently only worked with libc5 and not glibc, but that they were working on glibc support. I was impressed with both her helpful attitude, and the fact that she knew about glibc, which had only been available for two months.

A few days later Insure++ arrived at my house. The box contained a CD-ROM, a 10-page booklet with installation instructions and a 500-page user's manual. I had the software installed on the computer within five minutes, even though I had one minor problem with their installation script. I then called Parasoft to get a license key. I was very impressed with the salesperson who answered. After he gave me the key, he helped me create a $HOME/.psrc file, the startup file for Insure++, and he walked me through one of the examples included on the CD-ROM. Then he showed me a few features of the product and gave me his telephone extension and told me to call him if I had any problems.

Insure++ operates by taking your C or C++ source code and creating a new file which contains your code plus some automatically generated statements. The purpose of these statements is to analyze how your program is using memory, function calls and variables, so that potential problems can be found. Insure++'s analysis is extremely detailed. It knows when you use uninitialized variables or memory. It knows when there are no longer any pointers to allocated memory (leaks). It knows when you reference past the end of an array or structure. It knows when you call functions improperly. And it knows even more. Insure++'s analysis is also very robust. It can handle programs that use threads and programs that use memory obtained from files created by mmap or SysV shared memory objects.

Insure++ is also easy to use. Instead of compiling your program with gcc, you compile and link it with a program called insure. The insure compiler takes care of generating the modified source files, compiling them with gcc and then deleting them. It also does compile-time error checking. After the program is compiled you run it in the normal fashion, and it runs as normal, except that it is analyzing itself for errors. Errors found at compile or run time can be logged to a file, to stderr or to stdout, and error messages can be customized in order to be interpreted by programs such as Emacs. The default behavior is to send error messages to an X11-based program called Insra. Insra displays the error messages in an easy-to-understand manner, and it acts as an interface with your editor. Insra can also save the errors, allowing you to reload them and fix the problems later.

Most programs are not completely self-contained. Instead, they use code from system libraries like the C library or the X11 library. In order to fully check your program for errors, these libraries must be compiled with insure. Since most people have no interest in recompiling something like the X11 library, Insure++ comes with precompiled versions of several system libraries including libc, libm, libX11, libXaw, libXt and libdlsym. If you need to use a library that's not included with Insure++, and you can't or don't want to recompile it yourself, you can just link with the standard library. Insure++ will still be able to do some error checking of the library functions, but it will not be as detailed or complete as it would be if the library compiled with insure.

The first thing I did with Insure++ was to get out a Sunsite archive CD and start compiling programs. This proved to be an excellent way to learn how to use the product. All the programs I tried proved to be easy to compile. In each case, I simply overrode the appropriate variables on the make command line to cause the program to be built with the insure compiler and the -g debug flag. For example, this command works with many X11 programs:

make CC=insure CDEBUGOPTS=-g

When make invokes the first insure compiler, the compiler starts up the Insra GUI to display any problems Insure++ detects at compile time. All future invocations of the compiler talk to the existing Insra process so you don't have a new window popping up for each compiler process that runs.

After the program is built, you just run it as you always have. When it starts up, it sends messages about any bugs it finds to Insra. Insra will display the source code in which the error occurred and the stack trace when it occurred. If you click on an element in the stack trace, Insra will start up your editor in the appropriate file so you can fix the problem.

One thing you notice when running programs compiled with insure is that they run much slower. I wrote a few test programs to try to quantify this phenomenon. If a program sits in a tight loop, makes no function calls and does not access any arrays, it does not slow down at all. If a program spends most of its time calling functions or accessing arrays, it slows down by about a factor of 80. Most programs do call functions and access arrays, so you can expect to see a significant slowdown. A good rule of thumb is every second of user time the program usually takes is going to translate to a minute of user time with Insure++.

I had a few minor problems with Insure++, such as Insra not being able to find source code when that code was spread out in multiple directories. I was always able to resolve these problems quickly by referring to the manual, which is well-indexed and well written. In fact, it is written well enough that it is fun to read even if you are not trying to solve a problem.

At one point I thought I had found a bug in Insure++. I had written the following test program:

int main()
 {
        char *x = malloc(30);
        char z = x[1];
        char y = x[31];
        int zz;
        x[0] = 0;
        z += 3;
        free(x);
        zz = x[0];
        return z*z;
 }

Insure++ detected that I had accessed past the end of the x[]array when I initialized y and that I had used x[] after I had freed its memory. However, it did not detect that I had initialized z with an uninitialized member of x[] (i.e., x[1] had not been initialized). I was excited because this omission gave me a chance to try out Parasoft's technical support. I put the test program on one of my web pages so I could show it to the people at Parasoft. I then called up the salesperson who had provided my license key and told him I had found a bug. He transferred me to one of their programmers. I gave the programmer the URL where the code was, and he took a look at it while I was on the phone with him. He told me that by default Insure++ does not check to see if variables less than 4 bytes long are uninitialized. He then told me what to put in my .psrc file to change this. Then he gave me his e-mail address and telephone extension so I could contact him if I had any other problems.

I had three separate interactions with three different people at Parasoft, and each interaction was very positive. I figured that people doing reviews for magazines got VIP treatment, but I wondered how other people would be treated. I found someone at work who had used Insure++ at a prior job, and I asked him what he thought about Parasoft's technical support. He told me that he also thought it had been excellent—so much for my special treatment.

After I had gotten familiar with the basics of using Insure++, I decided to try some of the more advanced features. The first one I investigated was the interaction between gdb and programs compiled with insure. Insure does a good job of hiding the modifications it makes to the source code from the debugger. For the most part, you can't tell that anything is different. One thing that is different is that the program will call the function _Insure_trap_error whenever it detects a problem in your code. By setting a debugger breakpoint in this function, you can get the program to stop each time Insure++ finds a problem. Then you can use the debugger to examine your program's variables and find out why the problem occurred. I tried this on a few programs and found it to be a very useful feature. There are also other functions you can call from the debugger to get information about which location in the program memory was allocated for a variable and how much memory is currently allocated.

Another feature I classified as advanced has to do with programs that use their own memory managers. Insure++ knows about malloc, calloc, free, new, delete and other standard memory management functions. This allows it to do in-depth error checking when you use these functions. It is fairly common for programs to have their own memory managers which allocate large blocks of memory and dole it out themselves. In order to get detailed error checking for these programs, it is necessary to teach Insure++ about your memory manager. I did not do this myself, but the people evaluating Insure++ at work did, and they indicated it was a fairly straightforward task. It is even possible to teach Insure++ about functions that have nothing to do with memory management and have it verify arbitrary things about the state of your program each time the function is called.

Figure 1. Insra Window Showing Errors

Figure 2. Stack Trace at Memory Leak Location

By now, I felt I knew enough about Insure++ to do an example for this review. I selected a printed circuit layout program called pcb-1.3 in the apps/circuits directory of sunsite.unc.edu. (There is a newer version of pcb available.) This program consists of slightly over 600KB of source code, so it is a reasonably complex program, even though it is by no means huge. When compiling the program, Insure++ warned about several possible problems, but none of them looked too serious. I then started up pcb using the example data that came with it. I had no idea how to use the pcb software, so I just started clicking on things. (This is usually an excellent way to find bugs.) After a few minutes of playing with the program, I had six memory leaks and one NULL pointer evaluation listed in the Insra window (see Figure 1). At this point, I clicked on the error for line 1352 of the program action.c, and the screen shown in Figure 2 popped up. This shows the stack trace when the memory was allocated and when it was leaked. If you click on the red arrows, the indicated source code will be opened in your favorite editor. A quick look at the source code shows why Insure++ is complaining. In the function ActionSave on line 1290 of action.c, the program gets a file name from the user:

1289 case F_LayoutAs:
1290  name = GetUserInput("enter filename:", "");
1291  if (name)
1292  SavePCB(name);
1293  break;

This file name is contained in memory set by malloc, pointed to by the variable name, a local variable in ActionSave. Later, at line 1352, we leave ActionSave:

1349  break;
1350  }
1351 }
1352 }

In addition to several memory leaks, Insure++ found a bug in the program related to using NULL pointers. It complained about this line:

469 return(&LineSortedByHighX[Layer][index2]);

I couldn't see anything wrong with this line by looking at it, so I restarted the program in the debugger, using the _Insure_trap_error breakpoint I mentioned earlier. When the program stopped, I examined the expression and it turns out that LineSortedByHighX[Layer] is NULL. The reason the program does not crash on this line is that we are taking the address of LineSortedByHighX[Layer][index2] rather than trying to dereference it. Presumably, at some later time, some function will evaluate this address and crash. Insure++ makes it possible to fix the problem and prevent a crash.

As you might be able to tell, I think Insure++ is a great product. It finds programming errors better than any other product I have used, it runs under my favorite operating system, and Parasoft's technical support is excellent. There is one problem—it's expensive. The cheapest configuration you can buy is a 3-user-node locked license, which costs $1,995 per node. Nothing I do on my home computer is worth that much money, so I won't be buying a copy of Insure++ for myself. I suspect that most people who program for fun will not be buying a copy either. Who should buy Insure++ then? People, or more likely companies, who do professional software development. When you are paying programmers several thousand dollars a month and bugs cost big money to deal with, then the price begins to look more attractive. When you consider that Insure++ might enable you to ship your product earlier, it begins to look very attractive. If you develop software for a living, you need this product. Insure++ runs under several flavors of UNIX and Windows too. In fact, anyone who is in an environment where programming time is money should consider evaluating Insure++. It is an excellent product.