Features/StaticAnalysisOfPythonRefcounts

Static Analysis of Python Reference Counts

Summary

I've written a static analysis tool that can detect reference-counting errors made in Python extension modules written in C. We'll run the tool on all such code in Fedora 17 and make an effort to fix as many problems as time allows.

Owner

• Name: Dave Malcolm

• Email: dmalcolm@redhat.com

Current status

• Targeted release: Fedora 17
• Last updated: 2012-02-10
• Percentage of completion: 60%

The code works, and has found real bugs, but still contains bugs itself. It's only been run on a small subset of the Python code in Fedora.

Fixed items:

• the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
• wrote an automated script for running the tool on a mock build, and generating a triaged report on the issues found
• created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472

Major TODO items remaining:

• only run it on source files that include <Python.h>
• automatically run it on all code in Fedora and file bugs
• go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

Detailed Description

This is the continuation of the "Static Analysis of CPython Extensions" Fedora 16 feature.

Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.

Unfortunately, such wrapper code must manually manage the reference-counts of objects, and mistakes here can lead to /usr/bin/python leaking memory or segfaulting. There's also plenty of code out there that doesn't check for errors.

In Fedora 16, we shipped an initial version of a static analysis tool I've written (gcc-with-cpychecker), implementing some basic checks.

The latest version of the checker can now detect reference-counting bugs, along with paths through code that doesn't properly handle errors from the Python extension API, and I've already used it to patch some significant memory leaks.

Benefit to Fedora

We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugs. These typically manifest as memory leaks. The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).

Scope

My hope was to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).

Unfortunately it's not possible to get the signal:noise ratio good enough in time for Fedora 17 for that.

The plan now is to automate running it on all of the C extension modules in Fedora 17, and to analyze the results. Initially bugs would be filed against the tool itself (gcc-python-plugin), and I would then triage them; genuine bugs would be reassigned to the appropriate components, and I'd try to fix the high-value ones, sending fixes upstream. However, this is a large task, and I'm likely to need help from package owners and other Python developers. False positives would thus remain as bugs in the checker itself, and I'd work on fixing them.

Work to be done:

• there's a gcc-4.7 incompatibility that will need a couple of days to fix
• automate running it on all code
• go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

How To Test

It's not clear that we need this section; the feature covers a distro-wide bug-fixing push.

I *have* written an extensive selftest suite for the checker itself, which is run when it is built.

User Experience

Non-technical end-users of Fedora should see no difference (other than more a robust operating system).

For examples of the output from the checker, see: http://dmalcolm.livejournal.com/6560.html

Dependencies

This is implemented via a GCC plugin that embeds Python; the checker itself is implemented in Python.

Contingency Plan

Given that this "Feature" is essentially a bug-sweep (using a new tool), we'll do as much as we can by the deadline. Any that's been done is an improvement to Fedora, but if the amount doesn't look impressive, we can drop this as a feature.

Documentation

Upstream documentation: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html

Release Notes

(assuming we achieve this:) To prevent memory leaks, all of the Python extension modules in Fedora 17 have been run through a static analysis tool that can detect reference-counting bugs.

Comments and Discussion

• See Talk:Features/StaticAnalysisOfCPythonExtensions