Embedding Python in Your C Programs

The language of choice for large, high-performance applications in Linux is almost always C, or somewhat less often C++. Both are powerful languages that allow you to create high-performance natively compiled programs. However, they are not languages that lend themselves to runtime flexibility. Once a C/C++ application is compiled, its code is pretty much static. At times, that can be a real hindrance. For example, if you want to allow users of a program to create plugins easily that extend the application's functionality, you have to deal with complex dynamic linking issues that can cause no end of headaches. Additionally, your users will have to know C/C++ in order to extend the application, which severely limits the number of people capable of writing extensions.

A much better solution is to provide your users with a scripting language they can use to extend your application. With a scripting language, you will tend to have much more runtime flexibility, as well as shorter development times and a lower learning curve that will extend the base of users capable of creating extensions.

Unfortunately, creating a scripting language is very much a nontrivial task that easily could become a major portion of your program. Fortunately, you don't need to create a scripting language. With Python, you can embed the interpreter directly into your application and expose the full power and flexibility of Python without adding very much code at all to your application.

Including the Python interpreter in your program is extremely simple. Python provides a single header file for including all of the definitions you need when embedding the interpreter into your application, aptly named Python.h. This contains a lot of stuff, including several of the standard headers. For compiling efficiency, it might be nice if you could include only those parts of the interface that you actually intend to use, but unfortunately Python doesn't really give you that option. If you take a look at the Python.h file, you'll see that it defines several important macros and includes a number of common headers that are required by the individual components included later in the file.

To link your application to the Python interpreter at compile time, you should run the python-config program to get a list of the linking options that should be passed to the compiler. On my system, those are:

-lpython2.3 -lm -L/usr/lib/python2.3/config

So, how much code does it take to run the Python interpreter from a C app? As it turns out, very little. In fact, if you look at Listing 1, you'll see that it can be done in as little as three lines of code, which initialize the interpreter, send it a string of Python code to execute and then shut the interpreter back down.

void exec_pycode(const char* code)
{
  Py_Initialize();
  PyRun_SimpleString(code);
  Py_Finalize();
}

Or, you could embed an interactive Python terminal in your program by calling Py_Main() instead, as in Listing 2. This brings up the interpreter just as if you'd run Python directly from the command line. Control is returned to your application after the user exits from the interpreter shell.

Embedding the interpreter in three lines of code is easy enough, but let's face it, just executing arbitrary strings of Python code inside a program is neither interesting nor all that useful. Fortunately, it's also far from the extent of what Python allows. Before I get too deep into what it can do though, let's take a look at initializing the environment that Python executes within.

When you run the Python interpreter, the main environment context is stored in the __main__ module's namespace dictionary. All functions, classes and variables that are defined globally can be found in this dictionary. When running Python interactively or on a script file, you rarely need to care about this global namespace. However, when running the embedded interpreter, you'll often need to access this dictionary to get references to functions or classes in order to call or construct them. You also may find that you occasionally want to copy the global dictionary so that different bits of code can be run in distinct environments. For instance, you might want to create a new environment for each plugin that you load.

To get at the __main__ module's dictionary, you first need to get a reference to the module. You can do this by calling the PyImport_AddModule() function, which looks up the module name you supply and returns a PyObject pointer to that object. Why a PyObject? All Python data types derive from PyObject, which makes it a handy lowest-common denominator. Therefore, almost all of the functions that you'll deal with when interacting with the Python interpreter will take or return pointers to PyObjects rather than another more specific Python data type.

Once you have the __main__ module referenced by a PyObject, you can use the PyModule_GetDict() function to get a reference to the main module's dictionary, which again is returned as a PyObject pointer. You can then pass the dictionary reference when you execute other Python commands. For example, Listing 3 shows how you could duplicate the global environment and execute two different Python files in separate environments.

// Get a reference to the main module.
PyObject* main_module =
   PyImport_AddModule("__main__");

// Get the main module's dictionary
// and make a copy of it.
PyObject* main_dict =
   PyModule_GetDict(main_module);
PyObject* main_dict_copy =
   PyDict_Copy(main_dict);

// Execute two different files of
// Python code in separate environments
FILE* file_1 = fopen("file1.py", "r");
PyRun_File(file_1, "file1.py",
           Py_file_input,
           main_dict, main_dict);

FILE* file_2 = fopen("file2.py", "r");
PyRun_File(file_2, "file2.py",
           Py_file_input,
           main_dict_copy, main_dict_copy);

I'll get into the details of how PyRun_File() works in a little bit, but if you look carefully at Listing 3, you should notice something interesting. When I call PyRun_File() to execute the files, the dictionary gets passed in twice. The reason for this is that Python code actually has two environmental contexts when it is executed. The first is the global context, which I've already talked about. The second context is the local context, which contains any locally defined variables or functions. In this case, those are the same, because the code being executed is top-level code. On the other hand, if you were to execute a function dynamically using multiple C-level calls, you might want to create a local context and use that instead of the global dictionary. For the most part though, it's generally safe to pass the global environment for both the global and local parameters.

At this point, I'm sure you've noticed the Py_DECREF() calls that popped up in the Listing 3 example. Those fun little guys are there for memory management purposes. Inside the interpreter, Python handles memory management automatically by keeping track of all references to memory transparent to the programmer. As soon as it determines that all references to a given chunk of memory have been released, it deallocates the no-longer needed chunk. This can be a problem when you start working on the C side though. Because C is not a memory-managed language, as soon as a Python data structure ends up referenced from C, all ability to track the references automatically is lost to Python. The C application can make as many copies of the reference that it wants, and hold on to it indefinitely without Python knowing anything about it.

The solution is to have C code that gets a reference to a Python object handle all of the reference counting manually. Generally, when a Python call hands an object out to a C program, it increments the reference count by one. The C code can then do what it likes with the object without worrying that it will be deleted out from under it. Then when the C program is done with the object, it is responsible for releasing its reference by making a call to Py_DECREF().

It's important, though, to remember when you copy a pointer within your C program that may outlast the pointer from which you're copying, you need to increment the reference count manually, by calling Py_INCREF(). For example, if you make a copy of a PyObject pointer to store inside an array, you'll probably want to call Py_INCREF() to ensure that the pointed-to object won't get garbage-collected after the original PyObject reference is decremented.

Now let's take a look at a slightly more useful example to see how Python can be embedded into a real program. If you take a look at Listing 4, you'll see a small program that allows the user to specify short expressions on the command line. The program then calculates the results of those expressions and displays them in the output. To add a little spice to the mix, the program also lets users specify a file of Python code that will be loaded before the expressions are executed. This way, the user can define functions that will be available to the command-line expressions.

#include <python2.3/Python.h>

void process_expression(char* filename,
                        int num,
                        char** exp)
{
    FILE*       exp_file;

    // Initialize a global variable for
    // display of expression results
    PyRun_SimpleString("x = 0");

    // Open and execute the file of
    // functions to be made available
    // to user expressions
    exp_file = fopen(filename, "r");
    PyRun_SimpleFile(exp_file, exp);

    // Iterate through the expressions
    // and execute them
    while(num--) {
        PyRun_SimpleString(*exp++);
        PyRun_SimpleString("print x");
    }
}

int main(int argc, char** argv)
{
    Py_Initialize();

    if(argc != 3) {
        printf("Usage: %s FILENAME EXPRESSION+\n");
        return 1;
    }
    process_expression(argv[1], argc - 1, argv + 2);
    return 0;
}

Two basic Python API functions are used in this program, PyRun_SimpleString() and PyRun_AnyFile(). You've seen PyRun_SimpleString() before. All it does is execute the given Python expression in the global environment. PyRun_SimpleFile() is similar to the PyRun_File() function that I discussed earlier, but it runs things in the global environment by default. Because everything is run in the global environment, the results of each executed expression or group of expressions will be available to those that are executed later.

Now, let's say that instead of having our expression calculator execute a list of expressions, you'd rather have it load a function f() from the Python file and execute it a variable number of times to calculate an aggregate total, based on a number provided on the command line. You could execute the function simply by running PyRun_SimpleString("f()"), but that's really not very efficient, as it requires the interpreter to parse and evaluate the string every time it's called. It would be much better if we could reference the function directly to call it.

If you recall, Python stores all globally defined functions in the global dictionary. Therefore, if you can get a reference to the global dictionary, you can extract a reference to any of the defined functions. Fortunately, the Python API provides functions for doing just that. You can see it in use by taking a look at Listing 5.

#include <python2.3/Python.h>

void process_expression(int num, char* func_name)
{
    FILE*        exp_file;
    PyObject*    main_module, * global_dict, * expression;

    // Initialize a global variable for
    // display of expression results
    PyRun_SimpleString("x = 0");

    // Open and execute the Python file
    exp_file = fopen(exp, "r");
    PyRun_SimpleFile(exp_file, exp);

    // Get a reference to the main module
    // and global dictionary
    main_module = PyImport_AddModule("__main__");
    global_dict = PyModule_GetDict(main_module);

    // Extract a reference to the function "func_name"
    // from the global dictionary
    expression =
        PyDict_GetItemString(global_dict, func_name);

    while(num--) {
        // Make a call to the function referenced
        // by "expression"
        PyObject_CallObject(expression, NULL);
    }
    PyRun_SimpleString("print x");
}

To obtain the function reference, the program first gets a reference to the main module by “importing” it using the PyImport_AddModule("__main__") function. Once it has this reference to the main module, the program uses the PyModule_GetDict() function to extract its dictionary. From there, it's simply a matter of calling PyDict_GetItemString(global_dict, "f") to extract the function from the dictionary.

Now that the program has a reference to the function, it can call it using the PyObject_CallObect() function. As you can see, this takes a pointer to the function object to call. Because the function itself already exists in the Python environment, it is already compiled. That means when you perform the call, there is no parsing and little or no compilation overhead, which means the function can be executed quite quickly.

At this point, I'm sure you're starting to think, “Gee whiz, this is great but it would be a whole lot better if I could actually pass some data to these functions I'm calling.” Well, you need wonder no longer. As it turns out, you can do exactly that. One way is through the use of that mysterious NULL value that you saw being passed to PyObject_CallObject in Listing 5. I'll talk about how that works in a bit, but first there is a much easier way to call functions with arguments that are in the form of C/C++ data types, PyObject_CallFunction(). Instead of requiring you to perform C-to-Python conversions, this handy function takes a format string and a variable number of arguments, much like the printf() family of functions.

Looking back at our calculator program, let's say you want to evaluate an expression over a range of noncontiguous values. If the expression to evaluate is defined in a function provided by the loaded Python file, you can get a reference as normal and then iterate over the range. For each value, simply call PyObject_CallFunction(expression, "i", num). The “i” string tells Python that you will be passing an integer as the only argument. If the function you were calling took two integers and a string instead, you could make the function call as PyObject_CallFunction(expression, "iis", num1, num2, string). If the function has a return value, it will be passed to you in the return value of PyObject_CallFunction(), as a PyObject pointer.

That's the easiest way to pass arguments to a Python function, but it's not actually the most flexible. Think about it for a second. What happens if you are dynamically choosing the function to call? The odds are that you're going to want the flexibility to call a variety of functions that accept different numbers and types of arguments. However, with PyObject_CallFunction(), you have to choose the number and type of the arguments at compile time, which hardly fits with the spirit of flexibility inherent in embedding a scripting language.

The solution is to use PyObject_CallObject() instead. This function allows you to pass a single tuple of Python objects instead of the variable-length list of native C data items. The downside here is that you will need to convert native C values to Python objects first, but what you lose in execution speed is made up for in flexibility. Of course, before you can pass values to your function as a Python tuple, you'll need to know how to create the tuple, which brings me to the next section.

Python data structures are returned from and passed to the Python interpreter in the form of PyObjects. To get to a specific type, you need to perform a cast to the correct type. For instance, you can get to a PyIntObject pointer by casting a PyObject pointer. If you don't know for sure what the variable's type is, though, blindly performing a cast could have disastrous results. In such a case, you can call one of the many Check() functions to see if an object is indeed of an appropriate type, such as the PyFloat_Check() function that returns true if the object could indeed be cast to a float. In other words, it returns true if the object is a float or a subtype of a float. If you'd rather know whether the object is exactly a float, not a subclass, you can use PyFloat_CheckExact().

The opaque PyObject structure isn't actually useful to a C program though. In order to access Python data in your program, you'll need to use a variety of conversion functions that will return a native C type. For example, if you want to convert a PyObject to a long int, you can run PyInt_AsLong(). PyInt_AsLong is a safe function, and will perform a checked casting to PyIntObject before extracting the long int value. If you know for sure that the value you're converting is indeed an int, it may be wasteful to perform the extra checking—especially if it's inside of a tight loop.

Often, Python functions ask for or return Python sequence objects, such as tuples or lists. These objects don't have directly corresponding types in C, but Python provides functions that allow you to build them from C data types. As an example, let's take a look at building a tuple since you'll need to be able to do that to call a function using PyObject_CallObject().

The first step to creating a new tuple is to construct an empty tuple with PyTuple_New(), which takes the length of the tuple and returns a PyObject pointer to a new tuple. You can then use PyTuple_SetItem to set the values of the tuple items, passing each value as a PyObject pointer.

You should now have enough to get started with embedding Python scripts inside your own applications. For more information, take a look at the Python documentation. “Extending and Embedding the Python Interpreter” goes into more detail on going the other direction and embedding C functions inside Python. The “Python/C API Reference Manual” also has detailed reference documentation on all of the functions available for embedding Python in your program. The Linux Journal archives also contain an excellent article from Ivan Pulleyn that discusses issues for multithreaded programs that embed Python.

Resources for this article: /article/8714.