Features/StaticAnalysisOfPythonRefcounts

Static Analysis of Python Reference Counts

Summary

I've written a static analysis tool that can detect reference-counting errors made in Python extension modules written in C. We'll run the tool on all such code in Fedora 17 and make an effort to fix as many problems as time allows.

Owner

• Name: Dave Malcolm

• Email: dmalcolm@redhat.com

Current status

• Targeted release: Fedora 17
• Last updated: 2012-05-29
• Percentage of completion: 100%

The code works, and has found real bugs, but still contains bugs itself. It's been run on all of the Python code in Fedora, but doing so has sometimes uncovered bugs in the checker.

Completed items:

• the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
• wrote an automated script for running the tool on a mock build, and generating a triaged report on the issues found
• created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472
• only run it on source files that include <Python.h> (implemented in git; not yet in a tarball release)
• automated running it on all code in Fedora using mock, injecting the plugin

IN PROGRESS: I'm working through the builds, going through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

Detailed status can be seen via the tracker bug and via a status file covering both bugs filed and those SRPMs for which bugs have not yet been filed (with reasons)

Everything in Fedora 17 linked against libpython2.7:

• 74 bugs filed for src.rpms, where the checker found genuine problems (20%)
• 71 src.rpms not requiring a bug to be filed (19%)
• 78 src.rpms waiting on fix for C++ support (21%)
• 18 src.rpms waiting on better SWIG support (4%)
• 13 src.rpms waiting on better Cython support (3%)
• 117 src.rpms requiring other followup work (31%)

out of 370 total src.rpms (that link against libpython2.7)

Within the critical path:

• 12 bugs filed for src.rpms, where the checker found genuine problems (3%)
  • RHBZ #790973 NEW - Bugs found in python-krbV-1.0.90-4.fc15 using gcc-with-cpychecker static analyzer
  • RHBZ #790979 NEW - Memory leaks and crashers found in python bindings in rpm-4.9.1.2-12.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #790983 NEW - Segfault under low-memory conditions found in libxml2-2.7.8-6.fc16 using gcc-with-cpychecker static analyzer
  • RHBZ #791180 NEW - Bugs found in anaconda-17.8-1.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #791359 NEW - Bug found in deltarpm-3.6-0.7.20110223git.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #794989 NEW - Bugs found in libpwquality-1.0.0-2.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #794991 NEW - Memory leak in PyErr_SetTDBError() found in libtdb-1.2.9-14.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #800075 NEW - Memory leaks and possible crashers found in newt-0.52.14-2.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #800086 NEW - Bugs found in pyOpenSSL-0.12-2.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #800146 ASSIGNED - Bugs found in python-ethtool-0.7-2.fc16 using gcc-with-cpychecker static analyzer
  • RHBZ #800200 NEW - Bug found in yum-metadata-parser-1.1.4-6.fc17 using gcc-with-cpychecker static analyzer
  • RHBZ #809945 NEW - Bug found in python-markupsafe-0.11-4.fc17 using gcc-with-cpychecker static analyzer
• 4 src.rpms not requiring a bug to be filed (1%)
  • dbus-python-0.83.0-9.fc17: Only false positives
  • python-pycurl-7.19.0-9.fc15: Only false positives
  • pygpgme-0.2-2.fc17: Only in module initialization
  • python-nss-0.12-3.fc17: Only in module initialization
• 2 src.rpms waiting on fix for C++ support (0%)
  • libimobiledevice-1.1.1-5.fc17: FIXME: C++
  • pycryptopp-0.5.29-3.fc17: FIXME: C++
• 12 src.rpms requiring other followup work (3%)
  • libsemanage-2.1.6-2.fc17: FIXME: build.log has: error: File /builddir/build/SOURCES/libsemanage-rhat.patch is smaller than 13 bytes
  • cryptsetup-1.4.1-2.fc17: FIXME: checker got confused by PyObjectResult, and some tracebacks
  • gnome-python2-2.28.1-8.fc17: TODO
  • libtalloc-2.0.7-4.fc17: TODO
  • gdb-7.4.50.20120120-17.fc17: TODO
  • kernel-3.3.0-0.rc3.git5.1.fc17: TODO
  • python-2.7.2-18.fc17: TODO: this one will probably require special-casing
  • libselinux-2.1.9-7.fc17: TODO: appears to have failed to build
  • policycoreutils-2.1.10-21.fc17: TODO: appears to have failed to build
  • libdmtx-0.7.2-6.fc17: FIXME: tracebacks:
  • pyparted-3.8-3.fc17: FIXME: did not see rpmbuild -bb in build.log
  • pyliblzma-0.5.3-6.fc17: FIXME: 4 tracebacks during build

Outside of the critical path:

• 62 bugs filed for src.rpms, where the checker found genuine problems (16%)
• 67 src.rpms not requiring a bug to be filed (18%)
• 76 src.rpms waiting on fix for C++ support (20%)
• 18 src.rpms waiting on better SWIG support (4%)
• 13 src.rpms waiting on better Cython support (3%)
• 105 src.rpms requiring other followup work (28%)

Detailed Description

This is the continuation of the "Static Analysis of CPython Extensions" Fedora 16 feature.

Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.

Unfortunately, such wrapper code must manually manage the reference-counts of objects, and mistakes here can lead to /usr/bin/python leaking memory or segfaulting. There's also plenty of code out there that doesn't check for errors.

In Fedora 16, we shipped an initial version of a static analysis tool I've written (gcc-with-cpychecker), implementing some basic checks.

The latest version of the checker can now detect reference-counting bugs, along with paths through code that doesn't properly handle errors from the Python extension API, and I've already used it to patch some significant memory leaks.

Benefit to Fedora

We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugs. These typically manifest as memory leaks. The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).

Scope

My hope was to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).

Unfortunately it's not possible to get the signal:noise ratio good enough in time for Fedora 17 for that.

The plan now is to automate running it on all of the C extension modules in Fedora 17, and to analyze the results. Initially bugs would be filed against the tool itself (gcc-python-plugin), and I would then triage them; genuine bugs would be reassigned to the appropriate components, and I'd try to fix the high-value ones, sending fixes upstream. However, this is a large task, and I'm likely to need help from package owners and other Python developers. False positives would thus remain as bugs in the checker itself, and I'd work on fixing them.

Work to be done:

• there's a gcc-4.7 incompatibility that will need a couple of days to fix
• automate running it on all code
• go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

How To Test

It's not clear that we need this section; the feature covers a distro-wide bug-fixing push.

I *have* written an extensive selftest suite for the checker itself, which is run when it is built.

User Experience

Non-technical end-users of Fedora should see no difference (other than more a robust operating system).

For examples of the output from the checker, see: http://dmalcolm.livejournal.com/6560.html

Dependencies

This is implemented via a GCC plugin that embeds Python; the checker itself is implemented in Python.

Contingency Plan

Given that this "Feature" is essentially a bug-sweep (using a new tool), we'll do as much as we can by the deadline. Any that's been done is an improvement to Fedora, but if the amount doesn't look impressive, we can drop this as a feature.

Documentation

Upstream documentation: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html

Release Notes

(assuming we achieve this:) To prevent memory leaks, all of the Python extension modules in Fedora 17 have been run through a static analysis tool that can detect reference-counting bugs.

Comments and Discussion

• See Talk:Features/StaticAnalysisOfCPythonExtensions